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I. REAL PARTY IN INTEREST 

The subject patent application was assigned by inventor Irene V. Hu to Fernandez & 
Associates, LLP on June 8, 2004. This assignment was recorded on August 3, 2004 at 
Reel/Frame 015649/0423. A corrective assignment by inventor Irene V. Hu to co-inventor 
Dennis S. Fernandez was executed on June 8, 2004. This assignment was recorded on August 8, 
2005 at Reel/Frame 016343/0986. The subject patent application was assigned by Dennis S. 
Fernandez to Lot 3 Acquisition Foundation, LLC on May 16, 2007. This assignment was 
recorded on July 3, 2007 at Reel/Frame 019515/0553. Therefore, Lot 3 Acquisition Foundation, 
LLC is the real party in interest in the subject patent application. 
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II. RELATED APPEALS AND INTERFERENCES 

Prior to acquisition of the present application by Lot 3 Acquisition Foundation, LLC, a 
Notice of Appeal was filed August 24, 2006, and an amended Appeal Brief was filed November 
17, 2006 in the present application. However, the Examiner reopened prosecution in an Office 
Action dated March 12, 2007, so the appeal was not instituted. Thus, there are no related appeals 
and interferences. 
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III. STATUS OF CLAIMS 

Claims 24-33 and 39-53 are pending in the application, of which claims 24 and 31 are 
independent claims. Claims 24-33 and 39-53 are rejected under 35 U.S.C. § 103(a). Claims 1- 
23 and 34-38 were previously cancelled. All claims 24-33 and 39-53 are being appealed. 
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IV. STATUS OF AMENDMENTS 

No amendments were presented in reply to the Final Office Action dated November 19, 
2009. Thus, the claims in the Claims Appendix represent the claims Appellant believes to be 
currently pending. 
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V. SUMMARY OF CLAIMED SUBJECT MATTER 

Independent claim 24 is directed to a system that comprises "a communicator" 1 that 
receives "first data" 2 and "second data" 3 associated with an "object." 4 The first data is received 
from a "fixed detector," 5 and the second data is received from a "mobile target unit." 6 The 
mobile target unit comprises a sensor that detects the second data. 7 The mobile target unit "is at 
least one of: mounted in the object, mounted on the object, carried in the object, or carried on the 
object." 8 The system further comprises "a processor" 9 that correlates the first data and the 
second data to generate "object location information." 10 

Independent claim 3 1 1 1 is directed to a method that comprises receiving first data from a 
fixed detector and second data from a mobile target unit. The first data and second data are 
associated with an object. The mobile target unit is mounted in, mounted on, and/or carried on 
the object. The first data and the second data are correlated to generate object location 
information. 



1 See, e.g., p. 3, line 7 through p. 6, line 5; p. 8, lines 22-30; p. 12, lines 8-12; and FIGS. 1, 3. Citations 
are to page and line numbers from the application as filed for Serial No. 09/823,509. 

2 See, e.g., p. 7, line 1 through p. 8, line 4; p. 8, lines 22-30; p. 9, lines 9-18; p. 10, line 7 through p. 1 1 
line 27; p. 18, line 17 through p. 19, line 19; p. 29, line 17 through p. 32, line 26; and p. 33, line 27 
through p. 35, line 12. 

3 See, e.g., p. 7, line 1 through p. 8, line 4; p. 8, line 22 through p. 9, line 18; p. 10, line 7 through p. 1 1 
line 27; p. 18, line 17 through p. 19, line 19; p. 29, line 17 through p. 32, line 26; and p. 33, line 27 
through p. 35, line 12. 

"See, e.g., p. 2, lines 3-15; p. 3, lines 12-19; p. 11, line 18 through p. 12, line 6; p. 17, lines 1-8; p. 19, 
lines 21-29; p. 23, line 25 through p. 25, line 28; and p. 26, line 24 through p. 27, line 17. 

5 See, e.g., p. 5, line 27 through p. 8, line 30; p. 27, lines 1-17; p. 28, lines 4-15; p. 30, line 4 through p. 
31, line 10; and p. 34, lines 24-29. 

6 See, e.g., p. 5, line 27 through p. 6, line 29; p. 9, lines 1-5; p. 27, lines 1-17; p. 35, lines 1-8; and FIGS. 
1-2. 

7 See, e.g., p. 5, line 27 through p. 6, line 29; p. 9, lines 1-5; p. 27, lines 1-17; p. 35, lines 1-8; and FIGS. 
1-2. 

8 See, e.g., p. 9, lines 1-18; p. 1 1, line 29 through p. 12, line 12; and p. 13, lines 9-16. 

9 See, e.g., p. 3, line 2 through p. 6, line 5; p. 9, lines 20-27; p. 10, line 17 through p. 1 1, line 20; p. 13, 
line 24 through p. 14, line 22; and FIGS. 2-3. 

10 See, e.g., p. 10, line 17 through p. 1 1, line 20; p. 18 line 17 through p. 21, line 20; p. 22, lines 7-13; p. 
29, line 17 through p. 32, line 26; p. 33, line 1 through p. 35, line 8; and FIG. 4. 
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No means plus function or step plus function claims under 35 U.S.C. § 1 12, sixth 
paragraph are being appealed. 



1 See, e.g., FIG. 4 and support for similar elements shown with respect to claim 24 listed above. 



VI. GROUNDS OF REJECTION TO BE REVIEWED ON APPEAL 

A. Whether claims 24-32, 39-40, and 42-50 are unpatentable under 35 U.S.C. § 
103(a) over Everett, Jr., et al., U.S. Patent No. 5,202,661 ("Everett") in view of Hyuga, U.S. 
Patent No. 5,818,733 ("Hyuga"). 

B. Whether claims 24-33 and 39-53 are unpatentable under 35 U.S.C. § 103(a) over 
Moengen, U.S. Patent No. 6,373,508 ("Moengen"). 

C. Whether claim 41 is unpatentable under 35 U.S.C. § 103(a) over Everett, in view 
of Hyuga and Kitamura, et al, U.S. Patent No. 5,554,983 ("Kitamura"). 



VII. ARGUMENT 

Appellant respectfully requests that the Board reverse the Examiner's 35 U.S.C. § 103(a) 
rejections. 

A. Rejections under 35 U.S.C. § 103(a) over Everett in view of Hyuga 

Appellant addresses individual claims below, but initially, Appellant notes that "[t]he 
mere fact that references can be combined or modified does not render the resultant combination 
obvious unless the prior art also suggests the desirability of the combination." M.P.E.P. § 
2143.01(111) (emphasis in original). In fact, Appellant submits that Hyuga cannot and should not 
be combined with Everett in the manner the Examiner suggests. In this regard, Appellant 
reiterates that "[i]f proposed modification would render the prior art invention being modified 
unsatisfactory for its intended purpose, then there is no suggestion or motivation to make the 
proposed modification." M.P.E.P. § 2143.01(V) (emphasis added). Thus, it is significant that 
the modification the Examiner proposes from Hyuga would render Everett unsatisfactory for its 
intended purpose. 

Specifically, Appellant's argument with respect to the combination of Everett and Hyuga 
can be outlined as follows: 

1 . A secondary reference is not combinable with a primary reference if the 
proposed modification to the primary reference, using the secondary reference, would 
render the primary reference unsatisfactory for its intended purpose. See M.P.E.P. § 
2143.01 (V). 

2. The Examiner suggests modifying Everett such that the intruder, of which 
Everett's surveillance system monitors, carries the mobile target unit as disclosed in 
Hyuga, but this modification clearly renders Everett unsatisfactory for its intended 
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purpose (as will be discussed further below) ~ how could an intruder be expected to carry 
part of the surveillance system intended to detect his unauthorized presence? 

3. Because Hyuga would render Everett unsatisfactory for Everett's intended 
purpose, Hyuga is not combinable with Everett, and Appellant respectfully requests 
withdrawal of all rejections based on the combination of Everett and Hyuga 
1. Claims 24-26, 28, 30-32, 39-40, and 42-50 

Everett discloses a "system for detecting intrusion into a secured environment using both 
fixed and mobile intrusion detectors . . . The mobile sensors are mounted on one or more mobile 
platforms which selectively patrol throughout the environment and may be rapidly deployed to 
any region in the environment where a fixed intrusion detector detects a possible intrusion" 
(Abstract). "The sensor suite onboard the mobile robot contains multiple, high resolution 
sensors of different types that are automatically oriented towards the potential intruder" (Everett, 
column 2, lines 31-34). 

The following table illustrates some elements of Appellant's claims 12 which the 
Examiner asserts 13 read on Everett. 



Claim 31 



Everett 



fixed detector 



"fixed sensor system, 12 of fig. 1" 



object 



intruder 



first data associated with the object 



presence of the intruder 



mobile target unit 



"mobile sensor, 19 of fig. 2" 



second data 



presence of the intruder 



12 Claim 3 1 is used for illustration purposes. Similar elements may be found in claim 24. 

13 Quotations regarding Everett are from the Final Office Action, Nov. 19, 2009, pages 2-3. 
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With reference to this table, it becomes abundantly clear that Everett cannot disclose or 
contemplate each element of Appellant's claims, regardless of what reference is combined with 
Everett. Specifically, Appellant claims "wherein the mobile target unit is at least one of: 
mounted in the object, mounted on the object, carried in the object, or carried on the object" 
(claim 31). Rewording this element using the table above results in "wherein the [mobile sensor] 
is at least one of: mounted in the [intruder], mounted on the [intruder], or carried on the 
[intruder]." 

Appellant's respectfully assert that the Examiner's rationale for this reading of claims is 
perplexing to say the least. How can an intruder be expected to carry (in himself or on himself) 
the very object that is secretly detecting his presence? Would a security guard hand the intruder 
the mobile sensor when the intruder illegally enters the building? In any case, Everett's intruder 
could simply dispose of the sensor, thus rendering Everett's system inoperable. 

These questions highlight Appellant's arguments that Appellant's claims are not obvious 
in light of the combination of Everett and Hyuga. For a claim to be obvious in light of a 
reference or combination of references, there must be some motivation to combine the references 
or make the suggested modification. 14 Otherwise, one of skill in the art would not look to the 
reference or combination of references to make the suggested modification. 

Here, no one of skill in the art would look to Everett when developing a system where an 
object being detected actually carries part of the detection system. One of skill in the art would 
know that an intruder could not be expected to carry a sensor that would give away his intrusion. 
For these reasons, Everett teaches against "wherein the mobile target unit is at least one of: 
mounted in the object, mounted on the object, carried in the object or carried on the object" as 
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recited in claim 31, and as similarly recited in claim 24. In fact, modifying Everett in this 
manner would render Everett unsatisfactory for its intended purpose. 15 

Thus, there is no motivation to modify Everett in this manner, regardless of what 
reference discloses a sensor carried by a sensed object. Even though Hyuga may disclose that an 
"on- vehicle imaging device 27c may be placed atop a golf cart 29" (Hyuga, column 5, lines 6-8), 
there is no suggestion or motivation to combine Hyuga ~ or any other reference — with Everett 
in the manner suggested by the Examiner. Therefore, regardless of what Hyuga discloses, 16 
Everett cannot disclose or contemplate, alone or in combination with Hyuga, "wherein the 
mobile target unit is at least one of: mounted in the object, mounted on the object, carried in the 
object or carried on the object" as recited in claim 31, and as similarly recited in claim 24. Thus, 
Appellant respectfully requests that the Board reverse the rejections of claims 24 and 31 based 
on Everett. 

Claims 25-26, 28, 30, 32, 39-40, and 42-50 variously depend from independent claims 24 
and 31. Therefore, Appellant asserts that dependent claims 25-26, 28, 30, 32, 39-40, and 42-50 
are differentiated from the cited references for at least the same reasons stated above for 



14 "The mere fact that references can be combined or modified does not render the resultant combination 
obvious unless the prior art also suggests the desirability of the combination." M.P.E.P. § 2143.01(111) 
(emphasis in original). 

15 "If proposed modification would render the prior art invention being modified unsatisfactory for its 
intended purpose, then there is no suggestion or motivation to make the proposed modification." 
M.P.E.P. § 2143.01(V) (emphasis added). 

16 Hyuga generally discloses a group of cameras at a location (e.g., a golf course) where each of the 
cameras are capable of recording an image of an object (e.g., golfer) located in a particular section of the 
location (e.g., at the ninth hole green). One of the cameras is selected from a known, current location of 
the object being observed: "Based on the locational signals from receiver (21), camera controller 
component (26) selects camera (27i) ~ (27n), (27c) and controls panning and tilting of the selected 
camera" (Abstract). The location of the object is known, for example, by a signal from a "mobile unit 
which is in the possession of each [golf] player or caddie . . . [which] transmits signals revealing its 
location" (Hyuga, column 4, lines 14-36). "Therefore, according to the invention, it is easy to know the 
location of the sender of the signals and take his picture with an imaging device" (column 2, lines 7-9). 
Once the system in Hyuga determines an actual location of the object to be monitored, a camera in the 
actual location is selected to record an image of the object — i.e., the golfer. 
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differentiating independent claims 24 and 31, as well as in view of their own respective features. 
Appellant thus requests that the Board reverse the Examiner's rejection of claims 25-26, 28, 30, 
32, 39-40, and 42-50. 

2. Claim 27 

Neither Everett, nor Hyuga, alone or in combination, discloses or contemplates "wherein 
the object is a vehicle" as recited in claim 27. The object Everett senses is an intruder (Everett, 
Abstract), and Everett does not disclose that the intruder can be a vehicle. Rather, the Examiner 
asserts "the mobile target unit includes a vehicle" (Final Office Action, Nov. 19, 2009, page 3; 
emphasis added). But at the same time, the Examiner admits that the "object" in Appellant's 
claims is the intruder (see the table listed above). The Examiner appears to argue that the single 
"object" in Appellant's claims reads on both Everett's intruder and mobile target unit. Appellant 
respectfully submits that this claim construction is erroneous. Nowhere does Everett disclose or 
contemplate that the intruder is a vehicle. 

Similarly, although Hyuga discloses that an "on-vehicle imaging device 27c may be 
placed atop a golf cart 29" (Hyuga, column 5, lines 6-8), the "golf cart" is not the object being 
imaged. Rather, the imaging device captures an image of the golfer (Hyuga, column 7, lines 1- 
32). For at least these reasons, neither Hyuga nor Everett, alone or in combination, disclose or 
contemplate "wherein the object is a vehicle" as recited in claim 27 (emphasis added). Appellant 
therefore respectfully requests the Board reverse the rejection of claim 27. 

3. Claim 29 

Neither Everett nor Hyuga disclose or contemplate an "accelerometer" as recited in claim 
29. The Examiner merely shows the existence of a "processor" in Everett and asserts that 
"velocity control and acceleration/deceleration ramping are performed by processor 417" (Final 
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Office Action, page 5). In contrast, Appellant respectfully submits that there is a significant 
distinction between an "accelerometer" and a processor that instructs an object to accelerate. For 
example, a gas pedal in a car instructs a car to accelerate, but a gas pedal is not an accelerometer. 
Therefore, Appellant respectfully requests that the Board reverse the rejection of claim 29. 
B. Rejection under 35 U.S.C. § 103(a) over Moengen 
1. Claims 24-26, 28-33, 39-40, and 42-53 

Moengen discloses "a method for manipulation of a movable object displayed in a 
television picture, [where] the distance between the object and fixed basic positions is detected at 
a time t together with the distance between the object and a television camera in a known 
position" (Abstract; emphasis added). "Both the position detectors [D] and the television 
cameras [K] are positioned in a pre selected x,y,z coordinate system . . . The positions of both 
the position detectors [D] and the cameras [K] are precisely defined in the x,y,z co-ordinate 
system" (Moengen, column 5, lines 51-59; emphasis added). 

"Where mobile cameras ... are used, it will be possible to determine the camera 
positions by means of the position detectors [D]" (column 12, lines 28-31; emphasis added). 
"[T]he position of a mobile camera ... is determined by means of GPS and transferred to the 
production location" (column 16, lines 17-20). "Four position detectors [D] are preferably used 
to achieve unambiguous detection of the position of the natural object N. The position of the 
object N is thereby solely determined by distance measurements, i.e. by trilateration" (column 6, 
lines 39-43; emphasis added). "The detected distances are given to a computing module 2 which 
by means of trilateration calculates the positions x,y,z at different times t and thereby also the 
path of the object N on the basis of positions detected at the different times t" (column 7, lines 4- 
8; emphasis added). 
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Because "the position of [Moengen's] object N is . . . solely determined by distance 
measurements [of fixed position detectors D]" (column 6, lines 39-43; emphasis added) (see also 
column 5, lines 51-59), Moengen does not disclose or contemplate, alone or in combination with 
any of the cited references, "a processor configured to correlate the first data [from the fixed 
detector] and the second data [from the mobile target unit] to generate object location 
information" as recited in claim 24 (emphasis added), and as similarly recited in claim 3 1 . 

The Examiner asserts "the second data is received from a mobile target unit (note the 
mobile camera is used . . .)" (Final Office Action, page 7). Regardless of whether or not 
Moengen's camera receives "second data," it is clear from Moengen that nothing from 
Moengen's camera is used to "generate object location information." Moengen explicitly states, 
"the position of [Moengen's] object N is . . . solely determined by distance measurements" 
(Moengen, column 6, lines 39-43; emphasis added), which explicitly excludes using any image 
data received through the cameras to determine position or location. Appellant therefore 
respectfully requests that the Board reverse the Examiner's rejections of claims 24 and 31. 

Claims 25-26, 28-30, 32-33, 39-40, and 42-53 variously depend from independent claims 
24 and 31. Therefore, Appellant asserts that dependent claims 25-26, 28-30, 32-33, 39-40, and 
42-53 are differentiated from the cited references for at least the same reasons stated above for 
differentiating independent claims 24 and 31, as well as in view of their own respective features. 
Appellant thus requests that the Board reverse the Examiner's rejection of claims 25-26, 28-30, 
32-33, 39-40, and 42-53. 

2. Claim 27 

The Examiner argues that "Moengen further teaches the object is a vehicle ... and the 
mobile target unit . . . [is] at least one of: mounted in the object, mounted on the object, carried in 
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the object, or carried on the object" (Final Office Action, page 8). However, nowhere does 
Moengen disclose or contemplate "wherein the mobile target unit is at least one of: mounted in 
the object, mounted on the object, carried in the object, or carried on the object" as recited in 
claim 31, and as similarly recited in claim 24. 

Moengen simply refers to a possibility of using a mobile camera (see, e.g., Moengen, 
column 12, lines 28-3 1), but does not refer to the mobile camera being mounted in/on or carried 
in/on the object being monitored. In fact, all other cameras in Moengen are disclosed to not be 
mounted in/on or carried in/on the object being monitored (see, e.g., FIGS. 1 and 8). Therefore, 
there is no suggestion in Moengen that these mobile cameras would be mounted in/on or carried 
in/on the object being monitored (in fact, Moengen suggests the opposite). Appellant therefore 
respectfully requests that the Board reverse the Examiner's rejection of claim 27. 

3. Claim 41 

Moengen discloses an "object vector" and a "camera vector" (see Moengen, column 5, 
line 67 through column 6, line 11), but neither of these vectors is a "movement vector" as recited 
in Appellant's claim 41 . For example, the "picture axis or the optical axis in a camera K is 
represented by a vector called the camera vector" (column 5, line 67 through column 6, line 2), 
and the "connecting line between the lens centre in a camera K and the object N ... is called the 
object vector" (Moengen, column 6, lines 12-14). These vectors are used to determine a current 
position of the natural object 17 so that a "synthetic track can be generated in a television picture, 
the synthetic track thus being intended to represent the path of a natural object N in the television 
picture during a given period" (column 14, lines 12-16). Thus these vectors are not "movement 

17 "The data processing unit Q also comprises a manipulator module 4 for generating a synthetic object S 
which corresponds to the natural object N . . . Thus in the position X, Y in the camera's picture plane, via, 
e.g., a video generator ... a synthetic object can be created, representing the natural object N in its 
position X, Y in the picture plane in the camera K" (Moengen, column 7, lines 31-41; emphasis added). 
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vectors," but are simply vectors that relate a current position of the natural object to a vector 
associated with a camera. 

Furthermore, the data from the camera vectors and object vectors are used to generate 
this synthetic path that the natural object has already traveled: "[t]he primary object here is that 
the synthetic object S which represents the natural object N should at all times display the 
position and/or the movement of the natural object N, as it would be represented in the 
television picture at any time t" (Moengen, column 7, lines 43-47; emphasis added). Therefore, 
Moengen does not disclose or contemplate "determining a movement vector to predict a future 
location of the object" as recited in claim 41 (emphasis added). Appellant therefore respectfully 
requests that the Board reverse the Examiner's rejection of claim 41. 
C. Rejection under 35 U.S.C. § 103(a) over Everett in view of Hyuga and Kitamura 

Claim 41 depends from independent claim 31. Therefore, Appellant asserts that 
dependent claim 41 is differentiated from the cited references for at least the same reasons stated 
above for differentiating independent claim 31, as well as in view of its own features. Appellant 
thus requests that the Board reverse the Examiner's rejection of claim 41 . 
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D. Conclusion 

In conclusion, Appellant respectfully requests that the Board reverse the Examiner's 35 
U.S.C. § 103(a) rejections of all pending claims 24-33 and 39-53. 



SNELL & WILMER L.L.P. 

400 E. Van Buren 
One Arizona Center 
Phoenix, Arizona 85004 
Phone: 602-382-6376 
Fax: 602-382-6070 
Email: dbarker@swlaw.com 



Respectfully submitted, 





David G. Barker 
Reg. No. 58,581 
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VIII. CLAIMS APPENDIX 

1-23. (Canceled) 

24. (Rejected) A system comprising: 

a communicator configured to receive first data associated with an object and second data 
associated with the object, wherein the first data is received from a fixed detector configured to 
detect the first data, and wherein the second data is received from a mobile target unit 
comprising a sensor configured to detect the second data, wherein the mobile target unit is at 
least one of: mounted in the object, mounted on the object, carried in the object, or carried on the 
object; and 

a processor configured to correlate the first data and the second data to generate object 
location information. 

25. (Rejected) The system of Claim 24 wherein the communicator is further configured to 
receive a target unit location, the processor being further configured to determine whether the 
mobile target unit is within a range of the fixed detector. 

26. (Rejected) The system of Claim 24 wherein: 

the object location information comprises at least one of object trajectory information, 
object physical location information, or object speed information; and 

the fixed detector is configured to provide an image of the object. 

27. (Rejected) The system of Claim 24 wherein the object is a vehicle. 

28. (Rejected) The system of Claim 24, further comprising a database configured to maintain 
a plurality of current positions associated with at least one of a plurality of sensors, a plurality of 
mobile target units, or a plurality of objects. 
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29. (Rejected) The system of Claim 24 wherein the mobile target unit comprises an 
accelerometer configured to provide data indicative of movement of the object to facilitate 
generating the object location information. 

30. (Rejected) The system of Claim 24 wherein: 
the object is an identified good; 

the mobile target unit comprises a radio-frequency identification device; and 

the fixed detector comprises a camera for observing the identified good, to facilitate 

enabling the sensor and the fixed detector to provide corroborative surveillance of the identified 

good. 

3 1 . (Rej ected) A method comprising : 

receiving, from a fixed detector, first data associated with an object; 

receiving, from a mobile target unit, second data associated with the object, wherein the 
mobile target unit comprises a sensor configured to detect the second data, and wherein the 
mobile target unit is at least one of: mounted in the object, mounted on the object, carried in the 
object, or carried on the object; and 

correlating the first data and the second data to generate object location information. 

32. (Rejected) The method of Claim 3 1 , further comprising activating a second fixed detector 
in response to the object location information. 

33 . (Rejected) The method of Claim 3 1 wherein the second data comprises an object 
identifier, the method further comprising registering the object identifier in a database to indicate 
association with the object. 

34-38. (Canceled). 
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39. (Rejected) The system of Claim 24 wherein the second data comprises the target unit 
location. 

40. (Rejected) The method of Claim 3 1 , wherein the correlating the first data and the second 
data comprises determining compliance with a scheduled object activity. 

41 . (Rejected) The method of Claim 3 1 , wherein the correlating the first data and the second 
data comprises determining a movement vector to predict a future location of the object. 

42. (Rejected) The system of Claim 24 further comprising a plurality of detectors each 
having a corresponding observation range, wherein at least one of the plurality of detectors is 
selected to observe the object. 

43. (Rejected) The system of Claim 24 wherein the first data comprises at least one of an 
image of the object or an identifier associated with the object. 

44. (Rejected) The system of Claim 24 wherein the first data comprises a plurality of images 
of the object captured at different times. 

45 . (Rejected) The system of Claim 24 wherein the second data comprises at least one of an 
image of the object or an identifier associated with the object. 

46. (Rejected) The system of Claim 24 wherein the second data comprises a plurality of 
images of the object captured at different times. 

47. (Rejected) The system of Claim 24 wherein the object location information is determined 
at least in part based on a fixed detector location. 

48. (Rejected) The system of Claim 24 wherein the object location information is determined 
at least in part based on a mobile target unit location. 

49. (Rejected) The system of Claim 24, further comprising a movement module configured 
to activate a second fixed detector in response to the object location information. 
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50. (Rejected) The method of Claim 3 1 , wherein correlating the first data and the second data 
to generate object location information comprises determining at least one of a trajectory or a 
speed of the object. 

5 1 . (Rejected) The system of Claim 24, wherein the mobile target unit comprises a locator 
unit configured to determine the target unit location. 

52. (Rejected) The system of Claim 24, wherein the fixed detector is configured to be 
selected in response to the processor's correlation of the first data and the second data. 

53. (Rejected) The system of Claim 49, wherein the fixed detector is further from the second 
fixed detector than from a third fixed detector. 
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[57] ABSTRACT 
A system for detecting intrusion into a secured environ- 
ment using both fixed and mobile intrusion detectors 
includes a multiplicity of fixed intrusion detection sen- 
sors are each deployed at specific, fixed locations within 
the environment. The mobile sensors are mounted on 
one or more mobile platforms which selectively patrol 
throughout the environment and may be rapidly de- 
ployed to any region in the environment where a fixed 
intrusion detector detects a possible intrusion. A com- 
puter receives the outputs of the fixed and mobile sen- 
sors and is communicatively coupled to the mobile 
platforms. The computer directs the mobile platforms 
to travel through the environment along paths calcu- 
lated by the computer, calculates a sum of weighting 
factors associated with the output of each sensor, and 
fuses the sensor outputs so that the sum is uninfluenced 
by the traveling of the mobile platforms. The sum is 
compared to a reference whereby an output is provided 
when the sum exceeds the reference. An alarm system 
operably coupled to the computer provides an intrusion 
alert when the output received exceeds the reference. 
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cific level of confidence that an ac 
occurred. 

SUMMARY OF THE INVENTION 

The invention described herein may be manufactured 5 The present invention provides a system for detecting 

and used by or for the Government of the United States intrusion into a secured environment using both fixed 

of America for governmental purposes without the and mobile intrusion detectors. The invention employs 

payment of any royalties thereon or therefor. an optimal mix of fixed sensors, positioned at specific 

BACKGROUND OF THE INVFNTTON locations throughout the environment, and sensors 

JJAOKUKUUND Oh IHfc INVENTION , 0 mQunted Qn Qne Qr more mobUe mbQts ^ patr0 , the 

The present invention relates to the general field of secured environment. The outputs of the fixed and mo- 
intrusion detection systems for secure environments, bile sensors are fused by a computer-based system that 
and more particularly to a system that integrates the emulates the assessment functions of its human counter- 
outputs of both fixed and mobile intrusion detection part. The system knows at all times where the robots are 
sensors in order to provide intelligent assessments of the 15 located, the zones of coverage for the mobile sensor 
level of security of the environment. suites, and the resultant effect of the robot's presence or 

Most security systems utilize fixed sensors such as motion on fixed intrusion detection sensors viewing that 

motion detectors and/or video cameras positioned at same area. 

specific locations throughout the environment to be The concept of mobility provides an unpredictable 

secured. Such environments include buildings, military 20 pattern with respect to precise sensor location and ori- 

bases, warehouses, storage yards, factories, and even entation, making the job of casing a scenario for the 

homes. A very great disadvantage with video camera purpose of identifying blind spots rather difficult in- 

surveillance is that it requires an alert human to monitor deed. The random "patrols and the uncertainty in the 

a bank of video displays over long periods of time. If the mind of the intruder as to what the robot's response 

human becomes distracted or fatigued, he may not no- 25 might be upon detection adds a psychological advan- 

tice an intrusion. Another problem with such security tage to the deterrent function of the security system. An 

systems is that in order to be sensitive enough to detect important advantage of the present invention is that if 

a potential intrusion, they are subject to nuisance trips. the fixed sensors detect a potential intrusion, the com- 

This causes erosion in confidence in these systems. Fur- puter directs a mobile robot to the vicinity to investi- 

thermore, such systems do not provide sufficient resolu- 30 gate the situation further. 

tion to be able to identify the specific nature of an intru- The sensor suite onboard the mobile robot contains 
sion or its precise location. Further, fixed positioned multiple, high resolution sensors of different types that 
intrusion detectors are not capable of tracking an in- are automatically oriented towards the potential in- 
truder's position with any real accuracy. trader. Data obtained from the mobile sensors is used to 

If the environment to be secured is very large, it 35 determine the probability of an actual intrusion so that 

becomes economically difficult to provide adequate nuisance trips are minimized. If an actual intrusion has 

sensor coverage throughout the entire environment occurred, the mobile robots are directed by the com- 

with fixed sensors. Furthermore, the fact that the sen- puter to follow the intruder and report the intruder's 

sors are positioned at fixed locations makes them vul- position. All this occurs in realtime. If the confidence 

nerable to being neutralized by a determined intruder. 40 level of an intrusion exceeds a threshold, an intruder 

There exists a tradeoff point wherein it becomes more alert is provided. Thus, this invention obviates the de- 
cost effective to outfit a platform with numerous sen- manding and tedious vigilance of a human required by 
sors, and let it travel from zone to zone, as opposed to conventional security systems. 

installing, wiring, and protecting the fixed installation of A multiplicity of fixed intrusion detection sensors are 
these same sensors so that all areas within a space are 45 each deployed at specific, fixed locations within the 
effectively covered. This is obviously a function of the environment. The mobile sensors are mounted on re- 
conditions to be monitored and the unit cost of the bots which selectively patrol throughout the environ- 
appropriate sensors, as well as the number of areas and ment and may be rapidly deployed to any region in the 
geometric configuration of the space to be protected. environment where a fixed intrusion detector detects a 
Some provision must be made to preclude site vulnera- 50 possible intrusion. A computer receives the outputs of 
bility when the robot is not on station, or perhaps even the fixed and mobile sensors and is communicatively 
down for maintenance. For these reasons, a combina- coupled to the mobile robots. The computer directs the 
tion of both fixed and mobile sensors is likely to evolve mobile robots to travel through the environment along 
as the appropriate solution in most cases, but obvious paths calculated by the computer, calculates a sum of 
problems arise with the operation of a mobile platform 55 weighing factors associated with the output of each 
in an area protected by permanently installed motion sensor, and fuses the sensor outputs so that the sum is 
detectors, in that the motion of the robots will set off the uninfluenced by the traveling of the mobile robots. The 
fixed alarms. sum is compared to a reference whereby an output is 
Therefore, a need exists for a security system which provided when the sum exceeds the reference. An 
allows one or more mobile robots to operate in an area 60 alarm system operably coupled to the computer pro- 
protected by fixed intrusion detectors. Furthermore, a vides an intrusion alert when the output received ex- 
need exists for an intrusion detection system that is ceeds the reference. X-Y positional data and the acti- 
capable of detecting the presence of a potential intru- vated sensor corresponding to an identified intrusion is 
sion, then scrutinizing the potential intrusion with presented on a video display. 

greater resolution to ascertain the nature of the intru- 65 Data fusion is accomplished such that the robot's 

sion, then determining the probability of an actual intru- motion does not trigger a 'fixed' sensor alarm, and data 

sion to minimize nuisance trips, and then providing an from both 'fixed' and 'mobile' sensors can be collec- 

intruder alarm if the threat of intrusion exceeds a spe- tively assessed in calculating a composite threat score 
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from individual sensor weightings, so as to achieve a by assigning "weights" to data representing 
high probability of detection with a corresponding low detector outputs, 
nuisance alarm rate. 

The present invention also provides a method for 
operating a security system having fixed and mobile 5 
intrusion detectors in a secured environment, where the 
mobile intrusion detectors are mounted on one or more 
mobile robots, and each the fixed and mobile intrusion FIGS. 3A and 3B are a schematic diagram of the 
detectors has a coverage zone. This method includes alarm interface unit. 

the steps of: (1) initializing a mathematical model of a 10 FIG. 4 is a block diagram of the mobile robot, 
fixed intrusion detector map comprised of nodes each FIG. 5 is a functional block diagram of the sonar 
having an initial value; (2) determining the position of subsystem. 

the mobile robot; (3) initializing a mathematical model FIG. 6 is a perspective view of the autonomous vehi- 
of a mobile intrusion detector coverage map comprised cle used in conjunction with the present invention, 
of nodes having an initial value; (4) reading the outputs 15 FIG. 7 is a perspective view of the propulsion mod- 
of the fixed and mobile intrusion detectors; (5) deter- ule. 

mining which the fixed intrusion detector coverage FIG. 8 is a schematic electrical diagram of the multi- 
zones include the position of the mobile robot; (6) inhib- plexer portion depicted in FIG. 5 of the present ' 
iting data corresponding to the outputs of any of the tion. 
fixed intrusion detectors that have the coverage zones 20 FIG. 9 is a schematic electrical diagram of the trans- 
that include the mobile robot; (7) assigning a weight to ducer switching relay (Detail A) of FIG. 8. 
data corresponding to each mobile intrusion detector FIG. 10 are flowcharts of the function of the control- 
zone associated with a mobile intrusion detector detect- ling processor associated' with the sonar system of the 
ing a potential intrusion by: (8) examining a sequence present invention. 

trigger history of the mobile intrusion detectors in order 25 FIG. 11 is a block diagram of the acoustic detection 
to ascertain the presence of purposeful motion by the array. 

potential intrusion in any of the mobile intrusion detec- FIG. 12 is schematic diagram of the amplifier/detec- 
tor zones associated with a mobile intrusion detector tor circuit of the acoustic detection array, 
detecting the potential intrusion; (9) increasing the FIG. 13 is an example of an environment in which the 
weights calculated in step 7 for each mobile intrusion 30 present invention may operate. 

detector threat score by a predetermined amount for FIG. 14 illustrates a path and obstacles obstructing 
each of the mobile intrusion detector zones in which the the path of the vehicle within the environment pres- 
purposeful motion is detected; (10) determining a mo- ented in FIG. 13. 

bile intrusion detector threat score by summing the FIG. 15 is a three dimensional probability distribution 
weights determined in steps 7, 8 and 9; (11) correlating 35 plot showing the perceived location of nearby objects 
the outputs of the fixed and mobile intrusion detectors and obstacles within the environment illustrated in FIG. 
by identifying a maximum weighted sum corresponding 14. 

to the outputs of the fixed intrusion detectors detecting FIG. 16 is a flowchart of the Path (World) Planner 
the potential intrusion, and having an associated fixed program software. 

intrusion detector zone that intersects a mobile intrusion 40 FIG. 17 is a flowchart of the Find-Path program 
detector zone in which the potential intrusion has been software. 

identified by; (12) rotating and translating the activated FIG. 18 is a flowchart of the Expand program soft- 
mobile intrusion detector zones associated with the ware. 

mobile intrusion detectors detecting the potential intru- FIG. 19 is a flowchart of the Retrace program soft- 
sion into the mobile coverage map; (13) assigning a 45 ware. 

value to each of the mobile intrusion detector coverage FIG. 20 is a flowchart of the Execute Path program 
zones associated with the mobile intrusion detectors software. 

detecting the potential intrusion to a predetermined FIG. 21 is a flowchart of the Execute Segment pro- 
value; (14) correlating the fixed intrusion detector cov- gram software. 

erage map with the mobile intrusion detector coverage 50 FIG. 22 is a flowchart of the Map Sonar program 
map for each location within the mobile intrusion detec- Software. 

tor coverage zone which intersects the interior of one FIG. 23 is a flowchart of the Local Assessment Soft- 
or more fixed intrusion detector sensor zones associated ware. 

with a fixed intrusion detector detecting the potential FIG. 24 is flowchart of the Update Sensor Software 
intrusion to produce a weighted sum representing a 55 which is a subroutine of the software presented in FIG. 
probability of an intrusion by: (15) determining a maxi- 23. 

mum fixed intrusion detector threat score correspond- FIG. 25 is a flowchart of the Assess Threat Software 
ing to the activated fixed intrusion detectors; (16) deter- which is subroutine of the software presented in FIG. 
mining a composite global threat score by summing the 23. 

mobile intrusion detector threat score and the maximum 60 FIG. 26 is flowchart of the Adjust Weight Purposeful 
intrusion fixed sensor threat score; and (17) providing Motion Software which is a subroutine of the software 
an alarm signal to an alarm output device if the compos- presented in FIG. 25. 

ite global threat score is greater that the threshold FIG. 27 is a flowchart of the Calculate Global Zone 
value. Weight Software which is a subroutine of the software 

Another advantage of the invention is that it provides 65 presented in FIG. 25. 
geographic correlation of fixed and mobile sensor alarm FIG. 28 is a flowchart of the Adjust Weight Angular 
data. A further advantage is that it provides a high Sensor Fusion Software which is a subroutine of the 
probability of detection with a low nuisance alarm rate software presented in FIG. 25. • 
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FIG. 29 is a flowchart of the Calculate Threat Soft- Reconfigurable Video Line Digitizer And Method For 

ware which is a subroutine of the software presented in Storing Predetermined Lines Of A Composite Video 

FIG- 25. Signal, filed Feb. 28, 1990, incorporated herein by refer- 

FIG. 30 illustrates the intrusion detection coverage ence. 

zones of the mobile robot. 5 Referring still to FIG. 2, the video signal output from 

FIG. 31 illustrates a plan view of the secured environ- video motion detector system 12g is provided to video 

ment as presented on a video display and sensor output multiplexer (MUX) 24 and then is received by video 

data. recorder (VCR) 26 and displayed on video monitor 28. 

FIG. 32 is a flowchart of software for initializing the RF link 20 includes RF link 20a coupled to host corn- 
fixed sensor coverage map. 10 puter 14, RF data transmitter 206 1 mounted onboard 

FIG. 33 is a flowchart of software for performing mobile robot 18, and RF video transmitter 20*2, also 

threat assessment. mounted on mobile robot 18. The video signal provided 

FIG. 34 is a flowchart of software for inhibiting the by video link 20&2 is received by video receiver 20c, 

fixed sensors. directed to MUX 24, and optionally stored by VCR 26 

FIGS. 3SA and 35B present a flowchart of software 15 and displayed by video monitor 28. Alarm system 22 

for performing fixed and mobile sensor output correla- provides an alert signal when the outputs of fixed sensor 

tion. system 12 and/or sensor system 19 detect perturbations 

FIG. 36 is a flowchart of software for adjusting the within the secured environment that have a high proba- 

weights of the bits corresponding to the sensor outputs. bility of corresponding to an intrusion. Each of these 

DESCRIPTION OF THE PREFERRED 20 "* * **** 

EMBODIMENT DT 

The present invention provides an intrusion detection ' Fixed. Intrusion Detection System . 
system which synthesizes data provided by one or more Fixed intrusion detection system 12 includes a multi- 
groups of intrusion detectors mounted on mobile robots 25 plicity of sensors capable of detecting perturbations 
with data provided by a fixed intrusion detection system within the area to be secured that may correspond to 
in order to compile a composite intrusion threat level of intrusions caused by entry of unauthorized personnel or 
an environment which is to be secured. A composite to other conditions inimical to the security of the area, 
intrusion threat level exceeding a specified threshold as for example, fire. The sensors are positioned at fixed 
results in an intrusion alert. 30 locations throughout the area to be secured in order to 

An overview of the present invention is described provide a high probability of detecting any such dis- 

with reference to FIG. 1, where there is shown fixed turbances, as would be well known by those skilled in 

sensor system 12 which preferably includes a multiplic- this art. 

ity of different types of intrusion detectors that provide The outputs of the sensors which comprise intrusion 

outputs to host computer 14 through alarm interface 35 detection system 12 are received by alarm interface unit 

unit 16. The invention also includes one or more secu- 16 which routes them to host computer 14. Computer 

rity sensor systems 19 each mounted on a mobile robot 14 evaluates the data, and if an intrusion is detected, 

18 which patrols throughout the environment. Security outputs an intrusion alert via alarm interface unit (AIU) 

sensor systems 19 are communicatively linked to host 16 so as to enable alarm system 22. 

computer 14 via radio frequency link 20 Algorithms 40 

implemented by software in host computer 14 integrate Alarm System 

the outputs of fixed sensor system 12 and security sensor Alarm system 22 includes an annunciation device, 

system 19 in order to determine the composite intrusion such as a bell or siren, and a panel of indicator lights 

threat level. If the composite intrusion threat level ex- which show the status of the individual sensors of fixed 

ceeds a predetermined threshold value, an alert signal is 45 sensor system 12. Such light panels are well known and 

output from host computer 14 through alarm interface commonly employed in the security industry. Alarm 

unit 16 to enable alarm system 22. A human operator system 22 responds to the various signals provided by 

monitoring alarm system 22 then may appropriately alarm interface unit 16. 
respond. 

This invention also provides navigational control for 50 Alarm Interface Unit 

the mobile robots 18 which: 1) periodically identifies Alarm interface unit (AIU) 16 is intended to make the 

the location and orientation of the mobile robot, 2) plans present invention compatible with existing installation 

a path for the robot to follow from its present location security systems consisting principally of an alarm sys- 

to a specified destination, and 3) provides directions to tem 22 and a plurality of fixed sensors such as are de- 

the mobile robot so that the path is executed in a manner 55 picted in FIG. 2 as sensors 12„- g . AIU 16 provides an 

whereby the mobile robot avoids running into anything. interface to host computer 14 which allows the host 

A more detailed description of the present invention computer to ascertain the status (alarmed or clear) of 
is presented with reference to FIG. 2. Fixed sensor the individual fixed sensors 12„- g . In addition, AIU 22 
system 12 may include several different types of sensors, provides an output interface which allows host com- 
as for example, door closure sensor 12a, window clo- 60 puter 14 to relay this status information to alarm system 



'e area sensor 12c, break-beam 22. AIU also provides an output signal via the VCR 

sensor 12d, passive infrared sensor 12e, pressure mat control line which allows host computer 14 to start and 

12/ and video motion detector system 12g, comprising stop VCR 26, and an output signal via a video MUX 

video camera 12g\ and video motion detector 12g2- control line which allows control of video multiplexer 
Such sensors are well known and commercially avail- 65 24. 

able. However, in the preferred embodiment, video AIU 16 is described with reference to FIGS. 3A and 

motion detector 12g2 is of the type described in in U.S. 3B. AIU 16 has, by way of example, 16 inputs on each 

Pat. No. 5,034,817 (application Ser. No. 07/486,465), of two IC's 4067, 8 outputs from each of four IC's 
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74LS244, and is connected to the 8-bit parallel printer 
port of host computer 14. In order for AIU 16 to con- 
trol 32 inputs and 32 outputs, it uses five bits 
(DB0-DB4) from the standard 8-bit parallel PC printer 
port as address bits. Pin number, data bit, and function 5 
of each bit are listed in TABLE 1. Bit DB5 is used to 
determine if AIU 16 is reading or writing (input/out- 
put). The DB6 (DATA-OUT) bit outputs data, while 
the ACK (DATA-IN) bit is used for receiving data. 

TABLE 1 10 

FUNCTION 
OUTA0 
OUTA1 
OUTA2 

SELA 15 
SELB 
WR/RD 
DATA— OUT 
DATA-IN 



In order to send a data bit to one of 32 outputs, the 
WR/RD (DBS) line is set to a logic high. DB0-DB2 
(OUTA0-OUTA2) provide the lower three output 
address lines. Since the 74LS259 addressable latch only 
has three input address lines and 8 outputs, a dual 2-to-4 
74LS139 multiplexer is employed to allow four 
74LS259 chips to be accessed, as determined by appro- 
priate logic levels provided by DB3 (SELA) and DB4 
(SELB). Each output from the 74LS139 is connected to 
the ENABLE input of one of the four 74LS259 multi- 
plexer chips. The logic level of SELA and SELB thus 
determine which one of the four 74LS259 is enabled. 
OUTA0-OUTA2 then provide the necessary address to 
select the desired output port on the enabled 74LS259, 
whereupon] 3D AT A OUT determines the logic level of 
that selected port. Once an output port sta 
retain the same value even after another port is selected. 
The value of an output port can only be changed if its 
corresponding address is set on lines DB0-DB4, other- 
wise the port value remains the same. 

In order to read in an input from AIU 16, the 40 
WR/RD line (DB5) is set to a logic low. 
OUTA0-SELA are used as the lower four address 
lines. The logic level of SELB, connected to 74LS139, 
determines which of the two 4067 analog multiplexers is 
enabled. If SELA is low, then ports 1-16 are available, 45 
whereas if SELB is high, then ports 17-32 are selected. 

Programming required to control the operation of 
AIU 16 is implemented in host computer 14 and may be 
written in "C", as presented, by way of example, in 
Appendix 1. 50 



will 35 



Mobile Robot 
Referring to FIG. 4, mobile robot 18 is a mobile plat- 
form which includes local processor 402, propulsion 
module 416, sensor system 19, and collision avoidance 55 
system 450. Propulsion module 416 receives instruc- 
tions from host computer 1 that direct it along a particu- 
lar path. Collision avoidance system 450 provides data 
to host computer 14 via local processor 402 for indicat- 
ing the presence of obstacles that may obstruct the path 60 
of mobile robot 18. More detailed descriptions of each 
of these systems which comprise mobile robot 18 are 
presented further herein. Mobile robot 18 is capable of 
being navigated throughout the operating environment 
by instructions provided by host computer 14 through 65 
local processor 402. Techniques for navigating a mobile 
platform in the manner employed in the preferred em- 
bodiment of the present invention are well known by 



those skilled, in this, technology, and are incorporated 
into a number of commercially available units. [Refer to 
U.S. Pat. No. 4,811,228, Method of Navigating An 
Automated Guided Vehicle, incorporated herein by 
reference]. 

Local Processor 

Local processor 402 performs the following func- 
tions: 1) coordinates the operations of all of the systems 
of mobile robot 18; 2) receives high level instructions 
from host computer 14; 3) passes drive commands to 
propulsion module 416; 4) receives X-Y position and 
heading updates from processor 417 of propulsion mod- 
ule 416; 5) receives range and bearing data representing 
the area surrounding mobile robot 18 from processor 
532 (See FIG. 5); 6) checks for potential collision condi- 
tions; 7) sends stop commands to propulsion module 416 
if a collision is eminent; and 8) passes required positional 
and sonar information to host computer 14. 

Local processor 402 may be programmed to perform 
the above recited functions in a high level language 
such as "C", or in an assembly language, such as 6502, 
in accordance with well known techniques,, as set forth, 
by way of example, in Appendix 2. 

Propulsion Module 

Referring to FIGS. 6 and 7, mobility and dead-reck- 
oning position determination of mobile robot 18 de- 
pends on two degree-of-freedom, computer-controlled 
propulsion module 416 having motion directed by local 
processor 402 which is mounted on mobile robot 18. 
Local processor 402 provides output instructions to 
processor 417 (FIG. 2) of propulsion module 416 in 
response to data received from host computer 14 so that 
mobile robot 18 may follow a path calculated by host 
computer 14. Processor 417 is typically provided as a 
component of commercially available propulsion mod- 
ules of the type employed in the preferred embodiment 
of the present invention. 

Commands are passed by local processor 402 to pro- 
cessor 417 which controls propulsion module 416 over 
a serial or parallel bus as a series of hexadecimal bytes 
which specify: (1) the direction in which to move or 
pivot, (2) the velocity, and, (3) the duration (distance or 
number of degrees.) The functions of propulsion mod- 
ule 416 include executing movement commands re- 
ceived from local processor 402 and performing dead- 
reckoning calculations. In an example of the preferred 
embodiment, these commands are: 

Byte 1 - Type Motion Requested (00 to 07) 

00 - Move forward 

01 - Move reverse 

02 - Pivot left 

03 - Pivot right 

04 - Offset left 

05 - Offset right 
07 - Stop 

Byte 2 - Requested Velocity 

Upper nibble is the port motor velocity; 

Lower nibble is the starboard motor velocity. 

Byte 3 - Distance to Move (Inches) or, 

Duration of Pivot (Degrees) or, 

Amount of Offset (Tenths of Degrees) 

Velocity control and acceleration/deceleration 
ramping are performed by processor 417 on an interrupt 
basis, while the main code performs all dead reckoning 



Evidence Appendix 
Appellant's Brief, page 51 



5,202,661 

9 10 

calculations. Cumulative X and Y components of dis- that any range reading is less than some critical thresh- 
placement as well as current heading, 0, are passed up old distance (as for example, 18 inches in the preferred 
the hierarchy via local processor 402 at recurring inter- embodiment), indicative of an imminent collision be- 
vals so that host computer 14 knows the location of tween mobile robot 18 and an obstacle, then local pro- 
mobile robot 18 in order to integrate data from sonar 5 cessor 402 sends a "halt" command to processor 417 
system 435 into a world model which is constantly (FIG. 4) of mobile robot propulsion system 416, and 
being updated with new information. The programming informs host computer 14 of this action. The path plan- 
which enables local processor 402 to control propulsion ner, implemented in host computer 14, then calculates a 
module 416 is typically provided with commercially new path for mobile robot 18 to follow that avoids the 
available propulsion modules similar to the type de- 10 obstacle so that mobile robot 18 may proceed to the 
scribed above. Examples of commercial models of suit- predetermined location. Collision avoidance system 450 
able propulsion modules are the "LABMATE," manu- employs a multitude of pre-positioned ultrasonic trans- 
factured by Transitions Research Corporation, 15 ducers 536: (FIG. 6) that are individually activated in 
Great Pasture Road, Danbury, Conn. 06810, or the any desired sequence by processor 532, thus enabling 
"NAVMASTER" by Cybermotion, 5457 Aerospace 15 collision avoidance system 450 to obtain range informa- 
Road, Roanoke, Virginia, 24014. tion in any given direction within an arc centered at the 
Referring to FIGS. 6 and 7, by way of example, pro- front of mobile robot 18 that extends forward in a 120 
pulsion module 416 includes a pair (only one wheel is degree conical pattern. Such ultrasonic transducers are 
shown) of coaxially aligned wheels 422 which are each commercially available from the Polaroid Corporation, 
driven by separate motors 424 which enable propulsion 20 Still referring to FIGS. 5, 8 and 9, the preferred em- 
module 416 to be differentially steered by rotating each bodiment of collision avoidance system 450 includes 
wheel 422 by different amounts, both in angular dis- transducer array 536a, which may for example, consist 
placement and direction. Wheels 422,' may for example, of 7 active ultrasonic' ranging sensors ' 536,y where ,• 
have 8-inch rubber tires, which when coupled with equals 1 to 7, with individual 30 degree beam widths, 
motors 424, provide a quiet, powerful propulsion sys- 25 spaced 15 degrees apart in an arc around the front of 
tem with minimal wheel slippage. Passive casters 423 mobile robot 18, as shown in FIG. 6. Processor 532 
mounted to propulsion module 416 provide it with sta- receives commands from local processor 402 and is 
bility. Armature shafts 428 of motors 424 are each cou- operably coupled to multiplexer 534a having outputs 
pled to a high-resolution optical rotary shaft encoder that selectively and sequentially activate transducers 
426 that provides phase-quadrature, square-wave out- 30 536, in accordance with instructions provided by pro- 
puts, where each square-wave output corresponds to a cessor 532. 

specific increment of angular displacement of a wheel The details of multiplexer 534a are illustrated gener- 

422. By way of example only, in the preferred embodi- ally in FIGS. 8 and 9. The seven ultrasonic transducers 

ment, encoders 426 produce 200 counts per revolution 536, are interfaced to ultrasonic ranging module 548 

of armature shaft 428, which translates to 9725 counts 35 through 12-channel multiplexer 534a, in such a way that 

per wheel revolution. Commands from local processor only one transducer, 536,-, is fired at a time. The ultra- 

402 direct the kinematics of propulsion module 416, as sonic ranging module 548 may be a "Texas Instru- 

for example, heading, velocity, and acceleration. Pro- ments" ranging module, Model No. SN28827, as is well 

cessor 417 (FIG. 2) of propulsion module 416 provides known. The heart of multiplexer 534c is a 4067 analog 

host computer 14 with its instantaneously computed 40 switch shown in FIG. 4. Processor 532 thus "sees" only 

dead-reckoning position and heading which is calcu- one transducer 536/ at a time through ranging module 

lated by counting the number and discerning the phase 548 and multiplexer 534a, and the software of processor 

relationships of the square-wave outputs of each en- 532 merely executes in a loop, each time incrementing 

coder associated with each wheel 422. Power to operate the index which thus enables a specific transducer 536/ 

mobile robot 18 is provided by battery 430 in accor- 45 of transducer array 536a. 

dance with well known techniques. Ultrasonic ranging module 548, if implemented with 

Collision Avoidance System P ° lar0 * M °f el *° f SN28827, is an active time-of-flight 

3 device developed for automatic camera focusing, and 

Collision avoidance system 450 (see FIGS. 4-5) is determines the range to target by measuring elapsed 

used to detect obstacles in the path of mobile robot 18 50 time between the transmission of a "chirp" of pulses and 

and provide data to host computer 14 so that the path the detected echo. The "chirp" is of one millisecond 

planner program can endeavor to calculate a path duration and consists of four discrete frequencies trans- 

which avoids any detected obstacles. Referring to mitted back-to-back: 8 cycles at 60 kHz, 8 cycles at 56 

FIGS. 5, 8 and 9, collision avoidance system 450 in- kHz, 16 cycles at 52.5 kHz, and 24 cycles at 49.41 kHz. 

eludes processor 532, multiplexer 534a and transducer 55 To simplify the circuitry involved, all timing and 

array 536a. Processor 532 of collision avoidance system time-to-distance conversions are done in software on 

450 receives commands from local processor 402 and processor 532. Three control lines are involved in the 

provides ranges and bearings, detected by transducer interface of the ultrasonic circuit board 548 to processor 

array 536a, from mobile robot 18 to nearby detected 532. The first of these, referred to as VSW, initiates 

surfaces that may present obstacles in the path of mobile 60 operation when brought high to +5 volts. A second 

robot 404. When an obstacle is detected within 5 feet of line labelled XLOG signals the start of pulse transmis- 

mobile robot 18, host computer 14 updates the world sion, while the line labelled MFLOG indicates detec- 

model, as will be described further herein, using target tion of the first echo. Processor 532 must therefore send 

range information provided by processor 532 through VSW high, monitor the state of XLOG and commence 

local processor 402 to host computer 14. This range 65 timing when transmission begins (approximately 5 milli- 

information is combined With additional information seconds later), and then poll MFLOG until an echo is 

describing the heading, 0, of mobile robot 18 as well as detected or sufficient time elapses to indicate there is no 

X-Y position data. If local processor 402 determines echo. 
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Four input/output (I/O) lines from processing unit Referring to FIG. 10 and continuing the example, 
532 handle the switching function of ultrasonic trans- Subroutine PING then is called, which enables the 
ducers 5361,- by activating a 4067 analog switch 544. particular channel of analog switch 544 dictated by the 
The binary number placed on these I/O lines by the contents of the Y Register. The VSW control line is 
central processing unit 532 determines which channel is 5 sent high, which initiates operation of the ranging mod- 
selected by switch 544; all other channels assume a high ule 548 with the selected transducer. The software then 
impedance state. Referring to FIG. 9, each of the relays watches the multiplexer output XLOG for indication of 
576 and its associated driver transistor 572 (illustrated in pulse transmission, before initiating the timing se- 
FIG. 8 as Detail A) is substantially identical. Relay quence. The contents of the timing counter, represent- 
driver transistor 572 is biased into conduction by cur- 10 ing elapsed time, can be used to calculate range to the 
rent limiting resistor 543 via the active channel of ana- target. If this value ever exceeds the maximum specified 
log switch 544 in such a fashion such that only one ran 8 e of the subsystem, the software will exit the loop, 
transistor 572 per switch 544 is conducting at any given otherwise the counter runs until MFLOG is observed to 
time, as determined by the binary number present at the 8° h 'gh, indicating echo detection. Upon exit from the 
outputs of buffers 537, 538, 540, and 542. This conduct- 15 timin g Ioo P> the ran 8 e value for * at particular trans- 
ing transistor 572 sinks current through its associated ducer is aved m indexed storage, and Subroutine PING 
relay coil of relay 576, closing the contacts of relay 576. returns to the main program. 

This action causes one of the transducers in array 536 to ^ e Re «f er 18 ' hen incremented to enable the next 

be connected to and hence driven by the ultrasonic ranging module m the sequence, and Subroutine PING 

ranging module 548, when ultrasonic ranging module 20 » «^ 38 before. This process is repeated until 

548 is activated by central processing unit 532 as de- the * Re «l ster f^ 5 j he value ° f the endm « md . ex j 

scribed below signifying that all transducers in the sequence specified 

Three I/O lines carry the logic inputs to processor ^ the J oeal P roc j* s °r 402 have *«» activated 

532 from the ranging module 548 for XLOG and „ ^V- Processor 532 then request r^—n from the 

MFLOG, and from processor 532 to the ranging mod- 25 lo f P rOC " SOr 402 t0 tranSnut 3,1 the Stored ran « e 

ule 548 for VSW. Non-inverting buffer 568 is used to of-!? ■ 1 v i, un. i, i a j .t. 

.... . . . , . RS-232 serial link. When acknowledged, the ranges 

tngger sw.tch.ng transistor 562 upon command from s^^y dumped out the serial interface art 

central processing umt532 to initiate the firing sequence laced q b fte ^ m . p ^ indexe(J 

of ranging module 548. Resistors 558 and 560 along 3Q £ ^ completion, processor 532 checks to see 

with trans.stor 556 form an .averting buffer for the if a „ ew command has ^ sent down ^ 

XLOG signal which md.cates the ac ual start of pulse ran ^ ce> and then s the s * sin 

transm.ss.on. Resistors 552 and 554 along w.th trans.s- the appropriate starting and ending indexes . ^ the 

tor 550 form an inverting buffer for the MFLOG signal software ^ continuousIy m a repet itive fashion, se- 

which md.cates detection of the echo. A final I/O line 35 quentially activating the specifled ranging modu i es , 

from processor 532 actuates power switch 533, shown converting elapsed time to distance, storing the individ- 

m FIG. 5, to power down the circuitry when not in use ual resultS( ^ then fmally transmitting an range data at 

to save battery power. once to the local processor 402, which is thus freed 

A second parallel port on processor 532 is used to from al) associated overhead. Securitv Sensor System 
receive commands from local processor 402 which tell 40 Referring to FIGS. 4 and 6, security sensor system 19 
processor 532 to power up the ranging units, and then, is m0U nted to propulsion module 416 and includes a 
which sensors to sequentially activate. Commands may multiplicity of different types of sensors capable f de- 
be in the form of an eight-bit binary number represented tecting perturbations within the secured environment of 
in hexadecimal format, where the upper nibble repre- t he type associated with an intrusion. In the preferred 
sents the starting ID and the lower nibble the ending ID 45 embodiment, sensor system 19 includes acoustical de- 
fer the sequence. For example, the command $17 can be tectjon system Wa> vibration sensor 196, infrared mo- 
used to activate and take ranges using sensors #1 tion dete ctor system 19c, microwave motion detector 
through #7 sequentially. Each time through the loop, syst em I9d, optical motion detector system 19e, ultra- 
upon completion of the sequence, the stored ranges are sonic detector system 19/ video motion detector system 
transmitted up the hierarchy to the local processor 402 50 I2 gl , and acoustical monitoring system 19i The outputs 
over an RS-232 serial link, with appropriate handshak- 0 f these sensor are all provided to local processor 402 
ing. The sequence is repeated in similar fashion until and then ultimately to host computer 14 which pro- 
such time as the local processor 402 sends a new com- cesses the information provided by the various sensors, 
mand down, or advises central processing unit 532 to Video camera 12g\ is mounted on head positioning 
power down the ranging subsystem with the special 55 servo 19/ which is controlled by host computer 14 by 
command $FF. techniques well known by those skilled in this art. Host 

The software of processor 532 may, by way of exam- computer 14 provides control signals to head position- 
pie, be structured as shown in FIG. 10. When energized ing servo 19/ in order to adjust the visual field of view 
by the local processor 402, processor 532 does a power- of video camera 12gi, based on the perceived threat 
on reset, initializes all ports and registers, and then waits 60 location, as discussed further herein. The outputs of 
for a command. When a command is latched into the acoustical monitoring microphone 19/ and video cam- 
I/O port, a flag is set automatically that alerts processor era 12#i are provided to video transmitter 20&2- 
532, which then reads the command and determines the 

starting and ending identities of the transducers 536,- to Ultrasonic Motion Detection System 

be sequentially activated. The interface circuitry and 65 Ultrasonic motion detection system 19/ may be used 

ranging units are then powered up, via switch 533 (FIG. to scan the environment in which mobUe robot 18 oper- 

5) and the Y Register is set to the value of the first ates in order to identify a potential intrusion through 

transducer to be fired. changes in measured target distances as seen by one or 
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more sensors in the 24-element array. The system cre- 
ates a reference template, which consists of the two 
most frequently observed range values for each of the 
individual sensors in the array, and then compares sub- 
sequent readings to this template. The presence of an 5 
intruder within the system field of view will result in a 
range value which does not agree with the two possibil- 
ities recorded earlier in the reference template. The new 
range reading will correspond to the distance to the 
intruder, and the index (position) of the affected sensor 10 
within the 360-degree array will provide a relative bear- 
ing, which is then used by the host computer to plot the 
position of the suspected intruder on the map display. 
Referring to FIGS. 5, 8 and 9, ultrasonic motion detec- 
tion system 19/ includes processor 532, multiplexers 15 
5346 and 534c, and transducer arrays 5366 and 536c. By 
way of example, transducer arrays 5366 and 536c each 
may include twelve ultrasonic transducers which are 
mounted in a 360 degree circular pattern around the top 
of mobile robot 18 as shown in FIG. 6. For purposes of 20 
reference, the twelve ultrasonic transducers of array 
5366 may be referenced as ultrasonic transducers. 536,-, 
where i = 1 to 12; and the twelve ultrasonic transducers 
of array 536c may be referenced as ultrasonic transduc- 
ers 536,, where i=13 to 24. All ultrasonic transducers 
536, may be of an identical type. 

Ultrasonic motion detection system 19/ performs 
ranging in a manner that is virtually identical to the way 
in which collision avoidance system 450 operates. The 3Q 
only difference between the hardware implementations 
of the two systems is that ultrasonic detection system 
19/ includes two multiplexers and 24 ultrasonic trans- 
ducers, whereas collision avoidance system 450 em- 
ploys one multiplexer and seven ultrasonic transducers. 35 
Therefore, it is to be understood that the descriptions of 
multiplexer 534a and transducer array 536a, illustrated 
and described with respect to FIGS. 5, 8 and 9, also 
apply to multiplexers 5346 and 534c, and to transducer 
arrays 5366 and 536c. Processor 532 interacts with mul- ^ 
tiplexers 5346 and 534c in the same manner as processor 
532 interacted with multiplexer 534a. Furthermore, the 
data generated by ultrasonic detection system 19/ is 
provided through local processor 402 to host computer 
14. 45 

Optical Motion Detection System 
Optical motion detection system 19e may be of any 
suitable type of commercially available optical motion 
detector. An example of an optical motion detectors 50 
suitable for use in the present invention is that described 
in U.S. Pat. No. 4,902,887, "Optical Motion Detector 
Detecting Visible And Near Infrared Light", incorpo- 
rated herein by reference. Another type of optical mo- 
tion detector suitable for use in the present invention is 55 
manufactured by Sprague, Model No. ULN-2232A. 

Infrared Motion Detector System 
Infrared motion detector system 19c is, by way of 
example, similar in operation to the Sprague Model No. 60 
ULN-2232A, identified above, except that a wave- 
length of 10 micrometers is used. Examples of infrared 
motion detectors suitable for employment in the present 
invention include the passive infrared sensing system 
manufactured by Eltec Instruments, Inc., Model No.'s 65 
822, 826B and 4192-3, as well as passive infrared detec- 
tors manufactured by Linear Corporation, Series 6000, 
8000, and 9000. 
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Microwave Motion Detection System 
Microwave motions detectors such as employed in 
microwave motion detector system 19d are well 
known. Such detectors rely on the Doppler shift intro- 
duced by a moving target to sense the relative motion of 
an intruder. The electromagnetic energy associated 
with such detectors can penetrate hollow walls and 
doorways thereby allowing the sensor to "see" into 
adjoining rooms in certain circumstances. This operat- 
ing feature permits the sensor to check locked office 
spaces and warehouses without the need for actual 
entry into such places. 

Acoustical Monitoring Microphone 
Acoustical monitoring microphone 19/ is described in 
U.S. Pat. No. 4,857,912, "Intelligent Security Assess- 
ment System", column 6, lines 30-41, incorporated 
herein by reference. 

Vibration Sensor 
Vibration sensor 196 may be implemented similarly to 
acoustical monitoring microphone 19/ except that a . 
piezoelectric transducer, mechanically coupled to the 
structure of the robot 18, is used instead of a micro- 
phone such that the vibrations in the floor of the se- 
cured environment such as ma be caused by a potential 
intruder are coupled to the detection unit. 

Acoustical Detection System 
Acoustical detection system 19a is described with 
reference to FIGS. 4, 6, 11 and 12. This system provides 
bearing information to the source of detected noise and 
includes three acoustic sensor elements configured as 
shown in passive array 455, each operably coupled to an 
amplifier/detector circuit 457, shown in detail in FIG. 
11. The outputs of the three amplifier/detector circuits 
are provided to acoustic processor 458 which may be a 
6502 based, single board computer. Array 455 consists 
of three omnidirectional microphones 459 symmetri- 
cally oriented 120 degrees apart, and separated by a 
distance d, not shown. This system is mounted on top of 
robot 18 as shown in FIGS. 4 and 6. Sound traveling 
across array 455 triggers all three microphones 459 in a 
specific sequence dependent on the relative position of 
the acoustical source with respect to an array 455. Be- 
cause of the symmetrical orientation of microphones 
459, the firing sequence of comparators 339 associated 
with each microphone 459 is used to determine the 
bearing to the acoustical source through simple triangu- 
lation. Such passive acoustical surveillance schemes are 
well known by those skilled in this technology. 

Video Motion Detection System 
Video motion detection system 19g includes video 
surveillance camera 19h and video motion detection 
system 19g. The orientation of video surveillance cam- 
era 19h is controlled by head positioning servo 19/. 
Head positioning servos suitable for use in the present 
invention are well known and commercially available. 
Video motion detection system 19g is preferably of the 
type described in application Ser. No. 07/486,465, 
"Reconfigurabte Video Line Digitizer And Method 
For Storing Predetermined Lines Of A Composite 
Video Signal", filed Feb. 28, 1990, incorporated herein 
by reference. Video camera 19h is further positioned to 
bear on the intruder using refined analysis by the output 
of video motion detection system 19g if the intruder's 
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presence is detected by video motion analysis. Video tion. This growth is represented by the outer perimeter 
motion detection system 19g is temporarily disabled as 636 of operating environment 600. 
head positioning servo 19/ pans to reposition camera When entering data from collision avoidance system 
19h, and is re-enabled once the camera stabilizes. Soft- 450, seven ultrasonic ranging transducers 536/ (where 
ware which provides the video motion analysis used to 5 i= 1 to 7) in transducer array 536a are used, shown in 
orient camera 19ft is provided in Appendix 3, written by FIGS. 5, 6 and 8. If a given transducer 536 return (echo) 
way of example, in 6502 assembly language. shows that an object is within five feet of array 536a, the 

cell at the indicated location of the return is incre- 
Host Computer mented twice (up to a specified maximum). Also, the 

Host computer 14 performs the functions of building 10 probability value assigned to each of the eight neighbor- 
and maintaining the "world model", a mathematical ing ce l' s 'S incremented once, to partially take into ac- 
representation of the operating environment; perform- 00,1114 uncertainties arising from the 30-degree disper- 
ing path planning to generate the initial route of mobile sion an gl e of the ultrasonic beam generated by array 
robot 18; rerouting mobile robot 18 to avoid obstacles in 536fl - 

its path; formulating a composite intrusion threat score 15 In addition, each time a return is processed, all the 
based on data provided by fixed sensors 12 and mobile ceHs within a cone > which ma y b y wa y of example, be 
security sensor system 19; and providing an operator 10 degrees wide and five feet long (or less if an object 
interface. Host computer 14 may be, by way of exam- a PI>ears within five feet), have their assigned values 
pie, a 16-bit Intel 80386-based personal computer. Host decremented by 1. This erodes objects from the world 
computer 14 is programmed in a high level language 20 ma P that m no lon 8 er P resent S6tve * t0 refme the 
such as "C". By way of example, the more significant representation of existing objects as mobile robot 18 
source code program listings of software required to approaches. Objects are erased from the map at a 
operate the present invention are described below. slow .f ra ' e ™ an th ^ a 
These and other software implemented in the present „ mobile robot 18 tends to 
invention are identified in Appendix 1, herein. Program 25 ^obstructions 
listings for software identified in Appendix 1 are pro- An example of how data provided by collision avoid- 
vided in Appendix 2, herein. ance system 450 may be transformed into a mathemati- 

cal model of one example of an operating environment, 
World Model such as environment 600, is presented in FIG. 14, where 

D _ ... „ ..... , . 30 a path 638 from point A to point B, along which mobile 

Providing the capability of supporting autonomous ^* 18 (not shown) may be directed to follow, is ob- 

Zr^L rl^ *u a T Sltl0n K° f ™ ed by two obsicles 640. Other objects 642, 644, 

information regarding ranges and beanngs to nearby and ^ ^ ^ itioned whhin envir ^ nment goo. A 

objects, and the subsequent interpretation of that data in three dimensiona f; robability distributi on plot showing 

budding and maintaining the world model. The world 35 th e per ceived location ofnearby objects in environment 

map is a two-dimensional array of cells, where each cell m is il]ustrated m FIG . 15 ^ floor area of environ . 

m the array corresponds to a particular square having ment wa m is re p reS ented by cells ("nodes") 650. The 

fixed dimensions m the region being mapped. Free probability that any particular cel i is o^ed by ^ 

space is indicated with a cell value of zero; a non-zero object is proportiona , t0 the upward projection of My 

ceU value indicates the presence of an object. The cell 40 celI a]o the ^ ^ Techniques for creating world 

value represents the probability of a given square bemg maps of ^ operating environment or "motion area," 

occupied, which is useful when the precise location of suitable for use m the present invention are wel] 

an object is unknown. jjie world map contains positional information about 

The acquisition of range data is accomplished by use M the ^ own objects in the environment, 

of collision avoidance system 450. Target distance in- 45 Qne method for generating the initial world map is to 

formation is ultimately provided to host computer 14 doW nload data into host computer 14, where the data 

which assimilates the data into the world model while represents the operating environment, and is obtained 

mobile robot 18 is moving. Effective interpretation and fron) CAD drawings such as AutoCAD, a designing 

utilization of range data ts critical to achieve a reason- program by which any drawing, such as a building 

ably accurate representation of surrounding obstacles. 50 floorplan can be' reproduced in a microcomputer. A 

By using a simplified probability scheme and range second method is to manually input data into the host 

gating fixed arrays of sonar sensors, the mapping pro- computer where the data represents the coordinates of 

cess can take place in real-time while mobile robot 18 is the features of the environment using the MAPEDIT.C 

in motion. FIG. 13 is a plan view of an example of an subroutine (Appendix 1). A third method for generating 

operating environment 600, or "world," which for pur- 55 the world map is to have mobile robot 18 travel along 

poses of illustration, may be a two-room laboratory, its anticipated routes and use its sonar system 440 to 

where the perimeter of the environment is composed of generate data regarding the features of the environment 

wall segments 602, 604, 606, 608, 610, and 612. Further- that are then provided to host computer 14. Also, a 

more, by way of example to illustrate how the world combination of all three methods may be employed to 

map is constructed, environment 600 may also include 60 create or modify the world model, 

interior walls 614 and 616; doorway 618; book shelves The path planner operates on information stored in 

620 and 622; file cabinet 624; and tables 626, 628, 630, the world map to determine a route from the mobile 

632, and 634. The world map may be manually edited to robot's current position to its desired destination. The 

add additional features, such as hidden lines, doorways, basic search algorithm begins by "expanding" the initial 

etc. Each object in the world model then is automati- 65 cell corresponding to the mobile robot's current posi- 

cally "grown" by half the width of mobile robot 18 in tion in the floor map, i.e., each unoccupied neighbor 

order to model the mobile robot as a point during the cell is added to the "expansion list." Then each cell on 

Find-Path operation, described further within this sec- the expansion list is expanded. This process contiriues 
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until the destination cell is placed on the expansion list, Artificial Intelligence, Addison-Welsley, Reading, Mass., 

or the list becomes empty, in which case no path exists. 1984. In the Find-Path subroutine, a mathematical 

When a cell is placed on the expansion list, a value model of the operating environment, also referred to 

indicating the direction to the parent cell is stored in the the world map, is provided to this subroutine at step 

map array. Once the destination cell has been reached, 5 807. The environment is divided into a definite number 

retracing the path consists of merely following the di- of squares. The world map is implemented as a. two 

rectional arrows back to the source. During this pro- dimensional array of memory locations, and contains a 

cess, only those points representing a change in direc- byte for each square in the environment, such as a room, 

tion (an inflection point) are recorded. The entire path is where the size of a square can range from one square 

completely specified by the straight line segments con- 10 inch up to several square feet, depending on the desired 

necting these inflection points. Details of the path plan- map resolution and the size of the operating environ- 

ner are presented below. ment. 

Path Planner Next ' at step 808 tw0 s P ecial bytes .are stored in this 
memory array which represents the world map. One 
The path planner (See Appendix 1, PATHPLAN.C) 15 byte indicates the present location ("START") of mo- 
is implemented as a set of algorithms running on host bile robot 18; the second byte indicates the desired desti- 
computer 14 which enables mobile robot 18 to be di- nation ("DEST"). During the A* search process, the 
rected along a calculated path from the present position host computer 14 looks for the floor cell containing the 
of mobile robot 18 to a different position, where the DEST byte and similarly, during the backtrack process, 
positions are defined by Cartesian coordinates. Imple- 20 described below, the computer looks for the START 
mentation of the path planner is, by way of example, a byte. 

modification of techniques taught in: Winston, Patrick Next, at step 810, information about the source cell 
Henry, Artificial Intelligence, Addison-Wesley, Read- (such as X-Y location, cost, distance traveled, etc.) is 
ing, Mass., 1984. However, it is to be understood that put onto a "frontier" list which is a list of points on the 
the scope of the invention may include other implemen- 25 outer edge of the search envelope that are candidates 
tations of a path planner than those specifically pres- for the "expansion" process, described more fully be- 
ented herein. low. Putting the source cell on the frontier list "seeds" 
There are four basic tasks the path planner must ad- the path planner subroutine so that it has a cell to ex- 
dress in order to direct mobile robot 18 from point "A" pand. A loop is then entered at step 812 that terminates 
to point "B". They are described immediately below 30 only when there are no more cells on the frontier list or 
with reference to FIG. 16: a path has been found. If the frontier list is empty, then 

1. Finding a path to the destination (point B), hereaf- no path is possible and the search fails. 

ter referred to as the "Find-Path" operation at step The first step within the loop, at 814, is to find all the 

800. If no path exists, then this operation returns a cells on the frontier list with minimum cost, and then 

value of FALSE to the calling program. 35 put them on the expansion list at step 816. The "cost" of 

2. Retracing (or backtracking) the path found by the a cell is typically some computation of how "expensive" 
above "Find-Path" operation (discussed more fully it is for mobile robot 18 to travel to the location repre- 
further herein) to create a list of straight-line seg- sented by that particular cell. The actual cost function 
ments describing the route from source to destina- used in this implementation is described further herein, 
tion, where the source represents the present posi- 40 Next, all the cells on the expansion list are expanded 
tion (point A) of mobile robot 18. This operation is at step 818, as described more fully in the next section, 
performed at step 802. If the destination cell is reached, a path has been found 

3. Creating the movement commands which are ulti- and the algorithm terminates with a solution and returns 
mately directed to propulsion module 416 via local to the calling program from step 820. Otherwise, the 
processor 402 in order to execute the path. These 45 loop continues with the new frontier list (updated by 
operations are performed at step 804. the expansion process). 

4. If the path is successfully executed, then the path 

planner program returns a "successful" status from Expansion Subroutine 

step 806 to the calling program. Otherwise, the Referring to FIG. 18, the expansion process looks at 

program returns to step 800 in order to plan a new 50 all the neighbor cells of each cell on the expansion list, 

path. Each neighbor cell that is unoccupied and not on either 

Inability of mobile robot 18 to reach its intended the expansion or frontier list is placed on the frontier 

destination is usually attributable to obstacles or closed list. The details of this are provided below, 

doorways blocking the route. In that case, the planner A loop is entered at step 822 that terminates when all 

returns to Step 800 to try to find a different path, using 55 the cells on the current expansion list have been ex- 

the updated information now encoded in the model. panded. If no cells are left on the list, then a value of 

The path planner includes the following subroutines: FALSE is returned from step 824 to the path planner at 

Find-Path, Expansion, Backtracking, Path-Execution, step 818, indicating that the destination was not reached 

Segment-Execution, and Sonar-Mapping. These sub- during the current expansion and that further searching 

routines are described below. 60 of the updated frontier list is necessary. 

„. A D c , . t . The next cell (or first cell if this is the first time 

Find-Path Subroutine through the loop beginning at step 822) on the expan- 

As mentioned above, the Find-Path subroutine at step sion list is selected. First, a check is made to see if this 

800 is a set of algorithms which implement a modifica- cell can be expanded. The only cells that can be ex- 

tion of an A* search which is described below with 65 panded are those whose corresponding byte in the floor 

' reference to FIG. 17. The A* search is a type of search map array is equal to zero. If the value is not zero, this 

technique which is well known by those skilled in this cell may be occupied by an obstacle which has been 

art and which is taught in: Winston, Patrick Henry, detected by the robot's sensors. If so, then the value is 
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decremented at step 826 and the cell is put back onto the 
frontier list at step 828 to be expanded later. This tech- 
nique enables mobile robot 18 to travel a clear path in 
preference to a cluttered path, if a clear one exists. If no 
uncluttered path is found, mobile robot 18 may still be 5 
able to traverse the cluttered path. The capability of the 
expansion subroutine to determine alternative paths 
enables the robot to find a path even if sonar data pro- 
vided by sonar system 418 is somewhat faulty. 

If the contents of the current floor map cell are zero, 10 
then the cell can be expanded. Each of the cell's neigh- 
bors may be examined at steps 832, or 834 to see if any 
of the neighbors are occupied or unoccupied. "Neigh- 
bor" in this case refers to the four cells immediately 
adjacent to the current cell, i.e., located to the north, 15 
east, south and west of the current cell. These four 
"neighbors" may also be referred to as the "4-con- 
nected" neighbors. If the neighbor contains the special 
byte "DEST," then a path has been found at step 832, 
the X-Y location of the cell is saved at step 836, and a 20 
"TRUE" status is returned from step 838 to step 818 of 
the Find-Path subroutine. Otherwise, if the neighbor 
cell is unoccupied it is placed on the frontier list at step 
840. If it is occupied, it is ignored. 

Additionally, each cell has a "cost" associated with it. 
As in a typical A* search, at step 842, the cost is set 
equal to the distance traveled from the initial position of 
mobile robot 18 in order to get to the cell corresponding 
to the present location of mobile robot 18, plus the 3Q 
straight line distance to the destination cell. This is 
guaranteed to be less than or equal to the actual total 
distance from the source cell (present location) to the 
destination. This particular cost function tends to make 
the search expand in a direction towards the intended 35 
destination, thereby decreasing the search time. 

Finally, "arrow" information, used by the backtrack- 
ing subroutine, described below, is stored in the floor 
map cell corresponding to the current neighbor at step 
846. An "arrow" is one of four values indicating direc- 40 
tion, i.e., north, south, east, and west. The arrow indi- 
cates the direction to the neighbor's parent, which is the 
cell currently being expanded. 

Control is returned from step 840 to the top of the 
loop at step 822. 45 

Backtracking Subroutine 
Referring to FIG. 19, backtracking (also called re- 
tracing or segmentation) is a subroutine that creates a 
list of path segments which describe the desired path, 50 
based on the contents of the current floor map follow- 
ing the Find-Path operation, as described above. The 
procedure is very simple. Starting with the destination 
cell, the steps presented below are performed: 

1. Follow the arrow in the current cell to the next 55 
cell. Make the new cell the current cell. 

2. Return to the program that called the path planner 
if the new cell contains the value START, indicat- 
ing that a path to the destination has been found. 

3. Return to step 1, above, if the direction arrow of 60 
the current cell is the same as the direction arrow 
of the previous cell. 

4. Add the current X-Y coordinate to the path seg- 
ment list and update the segment counter. 

The output of the backtracking subroutine is a list of 65 
X-Y coordinates describing the "waypoints" through 
which mobile robot 18 must pass in order for the mobile 
robot 18 to reach the ultimate destination. 



Path Execution Subroutine 

Referring to FIG. 20, once a path segment list has 
been found, mobile robot 18 must then physically tra- 
verse the calculated path to reach the destination. Each 
segment of the path is executed individually in a loop 
beginning at step 860, whereby this process consists of 
having mobile robot 18 turn to the required heading and 
then having it travel in a straight line for a predeter- 
mined distance. 

Control is passed to the segment execution routine at 
step 870. A status condition is returned from step 871 to 
step 804 of the path planner, where the status condition 
indicates whether or not mobile robot 18 was able to 
successfully execute the segment. If it was successful, 
then the subroutine proceeds to step 860 where the next 
path segment (if any) is executed. Otherwise, an error 
condition is returned from step 871 to step 804 of the 
path planner. 

Segment-Execution Subroutine 
Referring to FIG. 21, during the execution of a sub- 
routine referred to as "Segment-Execution," the-plan? 
ner performs a number of tasks. First, step' 872 sends a 
command to propulsion module 416 to begin moving 
forward for a predetermined distance required by the 
path segment. Next, "Segment-Execution" enters a loop 
at step 873 which looks for status packets sent back by 
local processor 402. These consist of one or more of the 
following: 

1. A "move complete" report, indicating that propul- 
sion module 416 has finished moving the desired 
distance. If this occurs, an indication of successful 
status is returned by step 874 to step 870, illustrated 
in FIG. 20. 

2. An "obstacle" report, indicating that propulsion 
module 416 has stopped because an obstacle de- 
tected by collision avoidance system 450 impedes 
its path is returned by step 875 to step 870, illus- 
trated in FIG. 20. 

3. A "dead-reckoning" update. The present dead- 
reckoned position of mobile robot 18 is updated in 
the world map at step 876. 

4. A collision avoidance sonar packet is provided 
when sonar data is received by local processor 402, 
at which time the "sonar-mapping" subroutine, 
represented by the flowchart of FIG. 22, is invoked 
and the current representation of the world map is 
updated at step 878. 

The loop beginning at step 873 is repeated until either of 
the steps 874 or 875 within the loop is executed. 

Sonar-Mapping Subroutine 

Referring to FIG. 22 map sonar is a subroutine that 
receives the range information obtained by collision 
avoidance system 450 which then is used to update the 
local map. Although in the preferred embodiment, 
range information is obtained by use of ultrasonic trans- 
ducers 536/, other types of sensors could also be used to 
acquire such information, as for example, laser or opti- 
cal range fmders. 

One of the primary sources of errors with ultrasonic 
sonars is specular reflection. In order to reduce the 
number of erroneous sensor readings due to these types 
of errors, all detected ranges greater than specified 
distance, which may for example, be five feet, are ig- 
nored. Whenever a range reading is five feet or less, the 
value of the cell at the indicated range and bearing is 
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incremented twice (up to some maximum, as for exam- porated into the world model, representing the area 

pie, 16), and each of its 8-connected neighbors (all 4- under surveillance, as discussed in greater detail below, 

connected neighbors plus each of the diagonals) is in- The world model consists of a number of bit-mapped 

cremented once. parallel arrays, or layers, each indexed to an absolute 

During the execution of the map sonar subroutine, 5 X-Y grid representing the floor plan of the secured area 

sonar range returns or packets provided by processor (See Appendix 1: OA.C, MAPEDIT.C, and MAKE- 

532 through local processor 402 to host computer 14 are MAP.C). The floor plan is typically divided up into a 

processed and mapped. The sonar packets are decoded number of subset floor plans to achieve a realistically 

at step 880. Then a loop is entered at step 881 that con- sized model in order to facilitate near realtime raanipu- 

tinues until each range has been processed. At step 882, 10 lation of the encoded information. For any given subset 

the range is compared with five feet. If the range is floor P lan > tlle firs' layer 's devoted to X-Y positional 

greater than five feet, then processing proceeds to step information which is used for navigation and collision 

884. Otherwise, a transient obstacle will be added to the avoidance, as previously described, 

ma at step 883 by incrementing the appropriate cell Two additional parallel arrays, or layers, are assigned 

(indicated by the current range and bearing) by two, 15 to the world modeI ^ are used to represent the areas 

and each of the eight surrounding cells by one. This is of coverage of: (1) the fixed installation security sensors 

the manner in which transient obstacles are added to the for ^ portion of the floor plan in which robot 18 is 

map. In step 884, all of the cells in a ten degree cone positioned, and (2) the mobile security sensors mounted 

emanating from the location of the transducer out to the on robot 18 two lavers m referred to as the 

range return or four feet, whichever is less, are decre- 20 " flxed sensor coverage layer", and the "mobile sensor 

mented by one. In this way, transient obstacles that are coverage layer". 

no longer detected are gradually erased from the map. Slnce the P ur P° se of robot 18 IS l ° Patrol the secured 
environment, its position and orientation must be con- 
Collision Avoidance sidered when calculating the absolute world map repre- 
For a mobile robot to be truly autonomous it must 25 s^ 1 ' 0118 of the ^eas within the range of detection of 
cope with the classic problem of avoiding an unex- mobi , le system 19 ^ representations must be 
pected, unmapped obstacle. In the present invention, all translated j and r °* ted '" accordance with the robot's 
collision avoidance sensor information is statistically T'T V^l 1* ^VT"^ zones <f> 
represented in the world map, based on the number of „ J™ b f f f d w ' th * ose t frora the * nsor la ^ r f ° r 
times that an object was detected at a given cell location 30 th °^ S * nSOrS ^ ch f e tn «8 ered ' «? a 
[See APPENDIX 1: OA.C and CA.C], Conversely, m ° blk c ™f«™ actor representee of the max.- 
, , • , , , , , . . . ,' . . j rnum correlation between fixed and mobile sensors. 

f t* f kt, "° ° nger dete ° t < ed ™ s factor is used to calculate the composite 

at its original position, the probability of occupancy for threat 

the associated cell is decreased; if the probability is 35 In su tfle data fusion (im p leme „ t ed in 

reduced o zero the cell is again regarded as free space. software) monitors the mstantaneous state £ each of the 

Transient objects are added to the world map as they flxed md mobile securi Sfinsors tQ determine tne 

are encountered, and subsequently removed from the ence or absence of m ^ Securit Assess . 

model later if no longer detected at the same location. ment software indented in local processor 402 on- 

Since previously encountered obstacles are recorded in 40 ^ robot 18> rforms some . llocal „ cross correlation 

the world map, the mobile robot can avoid them at the of the outputs on sensor s tem 19 Resuhs are transmit . 

planning stage rather than during path execution. ted over Rp link 206] t0 host conjputer 14 for further 

A sample map created m this fashion is depicted in A hjghe r level ("global") correlation, imple- 
FIG. 15. Free space is represented by an array value of mented m host computer 14) then accounts for the out- 
zero and is shown in by the plane coincident with the 45 puts 0 f fixed sensor system 12, as well as the outputs 
X-Y plane. f rom however many mobile robots 18 are operating in 
Data Fusion System tne area - This ni 8 ner l evel correlation is referred to as 
Global Security Assessment (See FIG. 33). Each of 

The data fusion system, implemented in software these assessment functions, local and global, are dis- 

operatmg in host computer 14, integrates the outputs of 50 CUS sed in greater detail below, 
fixed security sensor system 12 and mobile security 

sensor system 19, to obtain a higher confidence solution, Loc al Security Assessment 
and to ensure that the motions of mobile robots 18 As previously stated, a number of different types of 
through the secured environment do not trigger a sys- sensors are preferably used onboard each mobile robot 
tem alarm. Referring to FIG. 23, the individual security 55 18 (as for example, infrared, microwave, ultrasonic, 
sensor values are updated as new sensor data becomes optical, video, sound, and vibration) to both increase 
available, whereupon a composite threat score is calcu- the likelihood of detection and decrease the frequency 
lated based on the updated sensor data. Data from both of nuisance alarms. FIG. 23 illustrates, by way of exam- 
fixed and mobDe sensors is used to calculate this global pie, a sample display 890 of video monitor 28 exhibiting 
composite threat score from individual sensor 60 the status of the various sensors. Techniques for pro- 
weightings in order to achieve a high probability of ducing such a display are well known by those skilled in 
detection with a corresponding low nuisance alarm this field of technology. The upper half of display 890 
rate. presents the state of each of the mobile sensors, the 
The data fusion system "knows" at all times where bearing to a possible intruder, and the current alarm 
each robot 18 is located, the zones of coverage for its 65 state. Sensors with a darkened background are tempo- 
onboard sensor system 19, and the resultant effect of its rarily disabled or unavailable. The lower half of display 
presence or motion on that portion of fixed sensor sys- 890 is used for displaying robot status information and 
tem 12 viewing the same area. This information is incor- current environmental conditions. 
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The motion detection sensors (depicted in the upper 
half of the display) are grouped into (24) zones. Each 
zone contains several different types of sensors. Local 
Security Assessment Software (Section 4.11 herein) 
performs a summation of weighted scores for all sensors 5 
within a particular zone, and calculates a composite 
threat score (Shown in the upper right of display 890; 
Refer to FIG. 31) which is proportional to the per- 
ceived threat presence. This software is implemented in 
host computer 14, and presented by way of example in 10 
Appendix 1. 

The Local Security Assessment Software detects 
patterns, such as purposeful motion across adjacent 
sensor coverage zones, and increases the associated 
composite threat accordingly. The system then acti- 15 
vates and positions secondary verification sensors on 
the robot as needed. At the same time, the current alarm 
threshold is dynamically calculated, based on the num- 
ber of sensor groups which are available, and other 
relevant conditions, such as ambient lighting, time of 20 
day, etc. The system classifies an alarm as an actual 
intrusion only when a complete evaluation has been 
performed using all sensor groups, and the resulting 
composite threat score exceeds the alarm threshold. 

The Local Security Assessment Software uses an 25 
algorithm which employs a polar representation of the 
sensor data to establish a composite threat score for 
each of 24, 15 degree, wedge-shaped zones about mo- 
bile robot 18, as shown in FIG. 25. The Local Security 
Assessment Software, provided by way of example in 30 
Appendix 2, is written in 6502 assembly language. The 
human operator is alerted to any situation where the 
composite threat score exceeds the specific alarm 
thresholds for a given zone, as discussed in more detail 
later. A threat assessment value in the range of 0-100 is 35 
provided as a quantitative indicator of classification 
confidence, and a threat vector originating from the 
robot's current position is graphically displayed by host 
computer 14 on video display 15. 

Reading in the Data 40 
On each pass through the Security Assessment Loop, 
(Refer to FIGS. 23-29) individual sensors which are in 
an alarm condition are identified by the functions 'up- 
date_range_sen' and 'update_onoff_ sen', (Refer to 45 
FIG. 24; Appendix 1: ACCESS.C) which create an 
array of output values that are then stored in the current 
information fields of the data structure. The baseline 
weighting values for this operation are taken from a 
two-dimensional array called 'init_ weight*. The preyi- 50 
ous values are stored in a history file and are pushed 
onto the top of a data structure called 'history —info' so 
as to provide an historical record of a finite period of 
time, as for example five minutes (See Appendix 1: 
ASSESS.C and ASSESS.H). 55 

Detecting Purposeful Motion 
The information stored in the history file is next ana- 
lyzed by the function *adjw_purposeful motion' (Refer 
to FIG. 26; Appendix 1: SUPPORT.C) for signs of 60 
purposeful motion, and the weights for affected sensors 
adjusted accordingly and stored in an intermediate data 
structure called 'inter_weights'. This is accomplished 
as follows: 

1. The algorithm identifies the first active sensor of a 65 
given group (i.e., sonar, infrared, microwave, etc.). 

2. Data stored in the history file is examined to deter- 
mine if adjacent sensors of the same group on either 
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side of the active sensor had previously been active 
within some prespecified period of time. 

3. If history of such activity is present, the weight of 
the active sensor is increased by an increment equal 
to its initial weight times some scaler SI. 

4. In the event an adjacent sensor is found to have 
been active, the history file is again examined to see 
if the next sensor in the array also had previously 
detected motion. 

5. If previous motion is again indicated, the weight of 
the active sensor is further increased by a second 
increment equal to its initial weight times some 
scaler S2. 

6. This process is then repeated for all other active 
sensors of the given type, after which the remain- 
ing groups of motion detection sensors are simi- 
larly examined in kind. 

In this fashion, if a temporal history of lateral motion 
across the field of view of the sensor array is present, 
such that adjacent sensors are activated in a distinct 
sequence, the resulting signature is classified as purpose- 
ful as opposed to random motion, and the active sensor 
weight is significantly increased. ' 

Most of the motion detector arrays (microwave, pas- 
sive infrared, acoustical, optical) are capable of angular 
resolution only, and provide no range information. An 
exception is the ultrasonic motion detection system 19/ 
(Refer to FIG. 5) which identifies a potential intrusion 
through changes in measured target distances as seen by 
one or more sensors 536, in the 24-element arrays 5366 
and 536c. This feature provides an additional level of 
analysis performed on sonar data accumulated in the 
history file, in that purposeful motion of an intruder 
should result in a somewhat continuous path target 
profile, with no significant discontinuities, or jumps in 
target position. 

Cross Correlation 
The next step in the local security assessment routine 
is referred to as "cross correlation" (See Appendix 1: 
SUPPORT.C). This involves checking for correlation 
among the different sensor groups (i.e., infrared, ultra- 
sonic, optical, etc.) to minimize nuisance alarms. The 
function 'adjw_angular_sensor fusion' (Refer to FIG. 
28) will increase the weight assigned to the output of a 
particular sensor if another type of sensor also detects 
motion along the same bearing, plus or minus some 
specified tolerance. This is accomplished as follows: 
1. The first active sensor of a given group is identi- 
fied. 

2 The current value of the sensor weight is incre- 
mented by the scaled weight of any confirming 
sensors of other types which view the same or 
immediately adjacent areas, and stored again in 
'current— weight', where: 



Weight Weight 



( same ^ ( adjacent ^ 
■« zone +d zone 
l^factor ) \ factor ) 



3. The adjusted weight values (from 'inter. weight') of 
the confirming sensors are used for this calculation, 
as opposed to the initial weights (from 'init.- 
weight'). 

4. In this fashion, the increase in weighting is propor- 
tional to the confidence factor of the confirming 
sensor. 
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5. This process is then repeated for all other active The preferred embodiment uses a bit-map approach 
sensors of the given type, after which the remain- which determines if any point in the fixed sensor cover- 
ing types are similarly examined. age layer lying within in a small area of ambiguity 
_ . _ _ , , . around robot 18 has the appropriate bit set for that 
Composite Threat Calculation 5 particular sensor . Referring to FIGS. 32 and 34, at the 

Once the various weight contributions have been beginning of each inhibition loop, the Security Assess- 

generated for the individual sensors of each type, the ment Software must mark all the sensors as uninhibited, 

function 'calc—threat' is called to sum the weighted then inhibit only those affected by the robot so that 

scores to generate a composite threat assessment for when a robot leaves the coverage area, the associated 

each of the 24 zones shown in FIG. 25. For each zone, 10 sensor will be enabled once again, 

there exists a predetermined alarm threshold value, and An alternative to the bit-map method is to examine 

any zone wherein the composite threat exceeds this each sensor coverage polygon to see if the robot is 

threshold is assumed to be in an alarmed condition. The within that polygon, then inhibit those sensors affected 

video camera Ugi will be energized by local processor by the robot's presence. The polygons would be prede- 

402 when the composite threat for any given zone ex- 15 fmed b y operator using the MAPEDIT.C software 

ceeds some scalar (typically 0.6) of the zone alarm presented in Appendix 1. The robot is modeled as a 

threshold. The axis of the most active zone is used to P° mt Consequently, representations of the fixed sensor 

graphically plot a threat vector on a map display of the coverage areas would be expanded by half the maxi- 

secured environment, presented on video display 15 mum dimension of the robot footprint. To determine if 

(FIG. 31). Head positioning servo 19/ (FIG. 4) then is 20 the robot 18 ca^B a fixed sensor to trip, the position of 

commanded to position video camera Ugi so that its the ro ** would be used as an index into the coverage 

optical axis is coincident with the orientation (bearing) *f y er - ^ ^ ue of that ^J"? *" the cove rage layer 

of this threat vector. Software for directing the head would .determine- which fixed sensors would be 

positioning servo 19/ is set forth, by way of example, in „ affected by the robot s presence. 

Appendix 3. Head positioning servo (panning) systems 25 A 1 n ° the K r * ^veto the bit-map approach could be 

such as employed in the present invention are well » Sed lf rob ° l » » ^f*™ f S ™ P , ST* ^ 

known, commercially available units. Software for im- < e -*- ^tangle). The Sutherland and Hodgrnan poly- 

plementing the display feature is provided by way of t ^ m& * lg0 " ^ . Ca " * U f d l ? . deteraune 

example, in Appendix 1 as INTRUDERS and IN- , n f wl f her or not ™ h } ot .".J? "^Tf T**? " Ft 

TMCT5 /- t a . .u .i. 30 tla ' lv or completely inside the defined region. [See 

DISP.C. In the event more than one zone is active, the Suth y erland) ^ £ y and Ho dgman, Gary W., "Reen- 

three zones with the highest composite threat scores tram Po , cli m CA g M , Vo l. 17, pp. 32-42, 

will generate threat vectors colored red yellow, and 1974] ^ ^ * chni cli a conca 7 e or con . 

white, m decreasing order of perceived threat seventy yex , £ (sensor covera ^ e * m ^ case) tQ 

Video camera 12*,, if energ.zed, will always be onented 35 an arbit convex , (robot) ^ semor ^ 

towards the direction corresponding to the zone havmg a{fected b the nce of the robot would be ^ 

the highest composite threat score, unless manually itedi ^ those not affected would ^ enabled If both 

overndden by a human operator. the coverage md robot polygons are convex, then the 

Global Security Assessment Cvrus a™ 1 Beck algorithm can be used. [See Cyrus, M. 

40 and Beck, J., "Generalized Two- and Three-Dimen- 

The Global Security Assessment Software addresses sional clipping," Computers & Graphics, Vol. 3, pp. 

two fundamental issues: (1) inhibiting those fixed instal- 23-28, 1978]. 

lation sensors which are momentarily activated by the In t ' he pre ferred embodiment, once the fixed sensors 

robot's passage through the protected area, and (2) affected by the robot have been inhibited, it is necessary 

fusmg the alarm status data from the fixed installation 45 t0 correlate data provided by all of the fixed sensors 

sensors with the data from the mobile sensors mounted with each otha< ^ well ^ ^ the outputs of mobile 

on robot 18 (or robots) m order to create a composite sensors on the robot. All sensors are modeled with 

representation of the perceived threat. Flowcharts for bit-maps [Refer to FIGS. 35A and 35B}. Data corre- 

software that would be implemented in host computer sponding to outputs of the sensors are integrated with 

14 for performing these functions are provided by way 50 the other maps to form the fixed-mobile correlation 

of example, in FIGS. 24-29 and 32-36. f act0 r. 

When robot 18 is moving in an area protected by An alternative method of correlating the data may be 

fixed installation sensors, the security assessment system employed if all the sensors are modeled solely with 

determines when the robot is about to enter or leave the polygons. Then it would be necessary to determine the 

coverage area of any given sensor. The sensor in ques- 55 specific polygon that is the largest subset of all the 

tion must be inhibited for that portion of the time when active polygons. Another alternative for performing 

robot 18 is moving through its field of view, so that the data correlation employs a hybrid scheme. In this 

motion of robot 18 does not generate a nuisance alarm. scheme, a hit bit-map can be created based on how 

Once the robot has departed the coverage area, that many polygons cover each cell in the map. Regardless 

particular sensor must be re-enabled. The robot's X-Y 60 of the method used, it is first necessary to rotate the 

position is used to obtain the identification of all fixed robot's coordinate system to correspond with that of 

sensors from the fixed sensor coverage layer that could the fixed sensors. 

be affected by the robot's motion at that particular in- When the robot is stationary, the data fusion software 
stant. These sensors are then gated out when construct- determines the degree of interaction of the outputs cor- 
ing the fixed-mobile correlation factor which is used to 65 responding to sensors on the robot with those of the 
calculate the global composite threat. Of course, the fixed sensors in the "hit" area in order to develop a 
robot's mobile coverage layer is time variant while composite threat weight. Accordingly, the system first 
mobile robot 18 is in motion. determines the fixed sensors which are active and their 
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associated coverage areas, as stored in the fixed cover- a subset of all other subsets. This is most easily accom- 
age layer. This information then is logically combined plished using the following steps: Pick an initial poly- 
with information provided from the calculated cover- gon (any one will do). Pick another polygon that has 
age areas, in absolute coordinates, of the active robot not been clipped. Clip the two polygons against each 
sensors. The result is stored as the fixed-mobile correla- 5 other, resulting in a third representing the intersection, 
tion factor, which is then used to calculate the global Using the resulting polygon as the initial polygon, re- 
composite threat. The fixed-mobile correlation factor peat the process until all polygons have been clipped, 
represents the maximum fixed intrusion detector threat The output is the maximum size polygon contained 
score corresponding to the fixed intrusion detectors that within all the other polygons. If all polygons are guar- 
detect a potential intrusion. The manner in which this 10 anteed to be convex, then the Cyrus and Beck clipping 
calculation is accomplished is discussed in the following algorithm can be used. Otherwise, each convex polygon 
paragraphs. must first be decomposed into two or more convex 
To initially set the appropriate bits in the fixed sensor polygons, then clipped. Alternatively, a more general 
coverage layer, the coverage extents are defined using algorithm capable of clipping a concave polygon 
the Map Editor (Appendix 1: MAPEDIT.C). A stan- 15 against another concave polygon could be used. [See 
dard polygon scan conversion algorithm [See Rogers, Weiler, Kevin, and Atherton, Peter, "Hidden Surface 
David F., Procedural Elements For Computer Graphics, Removal Using Polygon Area Sorting," Computer 
McGraw-Hill Book Company, New York, 1985] sets Graphics, Vol, 11, pp. 214-222, (Proc. SIGGRAPH 77), 
the correct bit for that particular sensor upon program 1977]. 

start-up, where an assigned bit corresponds to each 20 „„„„ . „ 

fixed sensor [Refer to FIG. 32 and Appendix 1: FIX- OPERATION OF THE INVENTION 

EDSEN.C]. As an example operational scenario, mobile robot 18 

The mobile sensor coverage layer associated with the ' is assigned a patrol route (or a discrete location) by the 

robot is created when needed to fuse data between the operator. The path planner calculates an appropriate 

sensors of fixed sensor system 12 and sensor system 19 25 path, whereby mobile robot 18 assumes its first assigned 

on robot 18. The information on robot heading and X-Y surveillance position. All primary detection sensors are 

position is first obtained from local processor 402, and online (optical, acoustical, infrared, vibration, and mi- 

then the mobile array values are generated according to crowave) and RF data link 20 is in standby operational 

preestablished rules described below. By way of exam- status. Video camera I2gi and associated RF video link 

pie, the preferred embodiment employed a 8-bit repre- 30 20&2 are deactivated. 

sentation. The position and orientation of the robot is If a possible disturbance is detected by one of the 
"frozen" before calculating the array values. Again, a fixed installation motion detectors of fixed sensor sys- 
standard polygon scan conversion algorithm (imple- tem 12, an appropriate signal is provided via AIU 16 to 
mented by POLYFILL.C, set forth in Appendix 1; See host computer 14. The human operator is alerted by an 
also Rogers, David F., Procedural Elements For Com- 35 audio beep from alarm system 22. In one type of sce- 
puter Graphics, McGraw-Hill Book Company, New nario, host computer 14 may determine that the trig- 
York, 1985) is used to set the appropriate bits for the gered sensor could not have been set off due to the 
X-Y locations in the array up to the imposed extents, motion of mobile robot 18 by noting that the current 
which are defined below. dead-reckoned position of the robot is not within the 

For example, the range of microwave motion detec- 40 designated coverage area of the alarmed sensor. The 

tor system 19d may be constrained by the room bound- Path Planner Software then dispatches robot 18 to a 

aries, as might be vibration sensor 196. Ultrasonic mo- location where mobile sensor system 19 can observe the 

tion detector system I9d may be constrained to where area in question. With no additional confirmation from 

all bits are set within a circle of ambiguity of some mobile security sensor system 19, the Realtime Assess- 

pre-specified diameter, centered at the reported range 45 ment Software, after a designated period of time, classi- 

value of the disturbance along the appropriate bearing. fies the threat as a nuisance alarm. All fixed and mobile 

The limits on an acoustical coverage area could be sensor reports are continuously time stamped and 

defined as a wedge of some specified angle of uncer- logged to disk in host computer 14 for later replay and 

tainty along the calculated bearing line, out to a dis- analysis. The robot then continues its assigned patrols, 

tance constrained by the room or map boundary. 50 In another example of a typical operational scenario, 

When a global correlation (correlating the outputs of a second disturbance is later detected by another sensor 

both triggered fixed and mobile sensors) is desired in a of fixed security sensor system 12. The robot is dis- 

static scenario, host computer 14 sets all the appropriate patched to the area and assesses the situation. In this 

bits in the mobile layer for sensor system 19 according example, however, the primary mobile detection sen- 

to the robot's current position and heading. [Refer to 55 sors also react in confirmation. The threat level is suffi- 

FIG. 35A]. In essence, this effects a coordinate trans- cient for the software to activate secondary sensors, and 

form which makes the mobile sensors look like fixed the ultrasonic motion detection system 19/ is enabled, 

sensors for that instant in time. Then, host computer 14 Cross correlation between sensors shows a strong likeli- 

detennines which sensors are alarmed, and begins to hood of an intruder at position (X,Y) on the map of the 

generate the fixed-mobile correlation factor using the 60 secure area, as indicated in the lower half of control 

sensor coverage information encoded in the fixed and screen 890 (FIG. 31) of video display 15. The software 

mobile sensor layers. The fixed-mobile correlation fac- which assembles this data and presents it on control 

tor is used as one of the components in calculating the screen 890 is provided, by way of example, in Appendix 

global composite threat. 1 as MAP.C, INDISP.C and INTRUDER.C. 

An alternative method of correlating the outputs of 65 The Realtime Assessment Software activates video 

both triggered fixed and mobile sensors is to find the camera 19h onboard the robot. The camera is automati- 

largest polygon completely contained within all the cally positioned along the bearing of the disturbance by 

other active polygons, i.e. find the largest subset that is head positioning servo 19/. The human operator is noti- 
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fied by a second audible alert, while video motion de- 
tector system 12g surveys the scene under surveillance. 
If the motion detected by video camera 12g\ confirmed 
as an actual intrusion, alarm system 22 provides an audi- 
ble output. The human operator is able to observe the 5 
intruder on video monitor 28, and may optionally see 
the (X,Y) position of the intruder depicted in a floor 
plan map, displayed on video display 15 by host com- 
puter 14. 

Once the relative bearing to an intruder has been 10 
established, the bearing can be used to calculate a mo- 
tion command which causes mobile robot 18 to rotate in 
place until the intruder is directly ahead of the robot, 
centered on the axis of the collision avoidance system 
450. Range information gathered by collision avoidance 15 
system 450, normally used to avoid an object in the path 
of the robot, provides information used to direct the 
robot towards the intruder and to follow him. The ro- 
bot's mean forward velocity is adjusted as a function of 
range to the intruder, and a calculated differential in left 20 
and right drive motor speeds is introduced as a function 
of how far off centerline the target appears. This differ- 
ential causes the robot .to turn towards the target being 
followed in a controlled fashion, until it appears cen- 
tered, all the while maintaining a specified distance 25 
interval. The robot's absolute X-Y position is graphi- 
cally displayed on a map of the secured area, while 
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camera transmits live video to display 890 for evalua- 
tion by the operator. 

Unless otherwise indicated all software is imple- 
mented in host computer 14. Program listings set forth 
in Appendix 1 are written in "C" and are provided by 
way of example only. Program listings set forth in Ap- 
pendices 2 and 3 are written in 6502 assembly language 
and are provided by way of example only. The software 
listed in Appendix 2 is implemented in local processor 
402. The software listed in Appendix 3 is implemented 
in the processor of video motion detector system 12gi. 
The scope of the invention includes all modifications to 
such software as would be evident in light of the ap- 
pended teachings. Furthermore, the scope of the inven- 
tion includes software written in languages other than 
those set forth herein. 

Obviously, many modifications and variations of the 
present invention are possible in light of the above 
teachings. For example, the scope of the invention in- 
cludes the employment of one or more mobile robots 18 
to patrol the secured environment. Therefore, singular 
references to mobile robot' 18 are to .be understood as 
referring to one or more mobile robots 18. Therefore, 
within the scope of the appended claims, the invention 
may be practiced otherwise than as specifically de- 
scribed. 



NAVY CASE NO. 72775 



APPENDIX 



Evidence Appendix 
Appellant's Brief, page 62 



31 


5,202,661 

APPENDIX 1 

SOFTWARE LISTINGS 
LANGUAGE: "C" 


32 


ANIMATE. C 


LISTS. C 


SYSCALL.H 


ASSESS. C 


LOGGER. C 


SYSDEFS . H 


ASSESS. H 


MAKEMAP . C 


TELE. C 


ASYNCH.C 


MAP.C 


TIMESTMP. C 


ASYNCH.H 


MAPCUR.C 


TIMESTMP . H 


BAUDRATE . C 


MAPDEFS . H 




BAUDRATE . H 


MAPEDIT.C 


voxi°H 


BB-EDIT.C 


MINTURNS . C 


V0X2.H 


BB-EDIT.H 




VOX3.H 


CA.C 


MYTYPES.H 


V.OX4.H .. 


CELLIST. C 




Z OOM • C 


DEADRECK.C 


PACKET . H 




DISPLIST.C 


PATH PLAN . C 




DISPLIST.H 






EXECUTE . C 


POLYFILL. C 




FILL1 . C 


PROTOS . H 




FIXED. C 


QUEUE . H 




FIXEDSEN.C 


RECAL. C 




FUNCKEYS . C 


ROBART . C 




FUNCKEYS . H 


ROBICONS . H 




GLOBAL. C 


RTCLK.H 




IBMIO . C 


SCRIPT. C 




IBMIO.H 


SENSOR. C 




IBMIO . C 


SENSOR. H 




IBMRTCLK.C 


SERIAL. H 




INDISP.C 


SETUP. C 




INTRUDER. C 






K2A.H 


SONPLOT. C 




LINESTUF.C 


SPEECH. C 




■ LINESTUF . H 


SQRT.C 




LINKDEFS . H 


SUPPORT. C 




LINKRBRT . C 


SUPPORT. H 
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We claim: 

1. A system for detecting intrusion in a secured envi- 
ronment, comprising: 
a first sensor fixedly positioned and operably disposed 
to provide a first output signal corresponding to 5 
detection of a first perturbation in said environ- 

a mobile platform operably disposed to be directed to 
travel along a path through said environment in , 0 
response to receiving path control output signals; 

a second sensor mounted to said mobile platform and 
operably disposed to provide a second output sig- 
nal corresponding to detection of a second pertur- 
bation in said environment; 

a digital data processor, communicatively coupled to 15 
receive said first output signal from said first sensor 
and to receive said second output signal from said 
second sensor, for assigning weighting coefficients 
to representations of each of said first and second 
output signals to generate weighted first and sec- 20 
ond values, for determining a sum of said weighted 
first and second values, for providing an intrusion 
alert output signal when said sum exceeds a refer- 
ence value, and for providing said path control 
output signals to said mobile platform, to direct 25 
said mobile platform to a particular location in said 
environment; 

an alarm system for generating an intrusion alert in 
response to receiving said intrusion alert output 
signal from said digital data processor. 30 

2. The system of claim 1 in which said first sensor is 
inhibited by said digital data processor when said mo- 
bile platform is in a zone of coverage of said first sensor 
and said first sensor is enabled by said digital data pro- 
cessor when said mobile platform is outside said cover- 35 
age zone. 

3. The system of claim 2 further including a data link 
communicatively coupled between said digital data 
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processor and said mobile platform for conveying said 
path control output signals from said digital data pro- 
cessor to said mobile platform and for conveying said 
second output signal from said second sensor to said 
digital data processor. 

4. The system of claim 3 wherein: 

said mobile platform further includes a collision 
avoidance system which generates a third output 
signal which is provided to said digital data proces- 
sor for indicating the presence of any obstacles that 
may obstruct said path of said mobile platform; and 

said digital data processor directs said mobile plat- 
form to travel a modified path to said particular 
location in said environment which avoids said 
obstacles. 

5. The system of claim 4 wherein said second sensor 
is an acoustical sensor. 

6. The system of claim 4 wherein said second sensor 
is a vibration sensor. 

7. The system of claim 4 wherein said second sensor 
is an infrared motion sensor. 

8.. The system of claim 4 wherein said second sensor 
is a microwave motion sensor. 

9. The system of claim 4 wherein said second sensor 
is an ultrasonic motion detector. 

10. The system of claim 4 wherein said second sensor 
is an optica] motion detector. 

11. The system of claim 4 further including: 

a head positioning servo mounted to said mobile plat- 
form and operably disposed to receive pan control 
output signals generated by said digital data pro- 
cessor through said data link; and 

a video surveillance camera mounted to said head 
positioning servo so that said video surveillance 
camera may be oriented to detect a video scene of 
a location at which said first or second perturba- 
tions were detected. 
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ABSTRACT 



When emergency button (11) of mobile unit (1) of a person 
who has suddenly fallen ill or other emergency, locational 
signals representing its own location and ID data are trans- 
mitted to receiver (21) of management unit (2) installed in 
the caddie master's office or at other locations. Based on the 
locational signals from receiver (21), camera controller 
component (26) selects camera (27j)~(27J, (27 c ) and con- 
trols panning and tilting of the selected camera, thereby 
adjusting its angle. Signals representing pictures taken by 
camera (27 \)~(27„), (27 c ) are received by visual image 
receiver component (28) and, together with vocal signals 
received from voice communication device (16) through 
voice communication device (24), transmitted from relay 
device (25) through a public circuit to visual image receiver 
component (32) and voice communication device (31) of 
remote unit (3). By operating the dial or push buttons of 
voice communication device (31), panning and tilting of 
camera (27j)~(27„), (27 e ) are controlled in order to adjust its 
angle, and vocal signals are transmitted from voice commu- 
nication device (31) to mobile unit (1). 

15 Claims, 10 Drawing Sheets 
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Fig. 6 
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Fig. 11 
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BACKGROUND OF THE INVENTION 

This invention relates to a method and system for com- 
munication of data including visual images taken by an 
imaging device. 

Conventionally, various systems for remotely controlling 
an imaging device in its focusing and imaging functions 
such as panning and tilting, and transmitting pictures 
obtained by said remote control through a communication 
means are widely known. 

Also known to those skilled in the art are a system 
disclosed in Japanese Patent Laying-open No. 191133/1988 
which calls for transmitting signals from a transmitting 
means attached to a subject to be photographed and taking 
photographs of this subject upon receiving these signals 
using a receiving means; another disclosed in Japanese 
Utility Model Laying-open No. 117625/1989 which calls for 
taking visual images of a subject by means of receiving 
signals transmitted from a transmitting means attached to the 
subject and causing an imaging device to follow the direc- 
tion of the strongest signal, and another disclosed in Japa- 
nese Patent Laying-open No. 134351/1981 which calls for 
following a subject using a transmitting means attached to 
the subject and, shooting for a specified period of time. 

However, the aforementioned system which controls per- 
formance of an imaging device by remote control is capable 
of receiving visual data signals only by manually changing 
the settings of the imaging device, and the configurations 
according to Japanese Patent Laying-open No. 191133/1988 
and Japanese Utility Model Laying-open No. 117625/1989 
merely call for causing an imaging device to follow a subject 
to take visual images thereof. Therefore, none is capable of 
fully utilizing the pictures taken. 

Further, in case of an emergency, for example, if condi- 
tions of a patient or his situation can be visually presented, 
it would help a doctor, who may be at a hospital or another 
remotely located facility in making an appropriate decision 
that might save the patient's life. For such a reason, there 
exists a need for a system which is capable of easily 
transmitting visual images of a specific subject to the doctor 
in such a case as above. 



In order to solve the problems in the prior art, an object 
of the invention is to provide a communication method and 
system for transmitting visual images which have been taken 
by an imaging device and sent by a sender to the receiving 
party through a relay station. 

A method according to the invention calls for transmitting 
signals representing the voice of a sender; receiving the 
vocal signals from the sender at a relay station; controlling 
imaging functions of an imaging device to take pictures of 
the sender; transmitting the vocal signals from the sender 
together with the visual images of same, which have been 
taken by the imaging device, from the relay station to a 
receiving party; and transmitting vocal signals of the receiv- 
ing party through the relay station to the sender. Therefore, 
as the invention is capable of transmitting visual images of 
the sender to the receiving party through the relay station 
and allowing 2-way voice communications between the 
sender and the receiving party through the relay station, the 



2 

receiving parly is able to easily obtain 
sender, and both parties can easily 

According to another feature thereof, the invention calls 

5 for transmitting data for the location of the sender so that the 
locational data is received and used to control the perfor- 
mance of the imaging device. Therefore, according to the 
invention, it is easy to know the location of the sender of the 
signals and take his picture with an imaging device. 

10 According to yet another feature of the invention, the 
receiving party controls the direction in which the imaging 
device is pointed. Therefore, it is easy for the receiving party 
to obtain desired pictures. 

According to yet another feature of the invention, physi- 

15 cal conditions of a subject are monitored; and the relay 
station transmits the data of his physical conditions to the 
receiving party together with his pictures taken by the 
imaging device. With the configuration as above, the receiv- 
ing party is able to easily obtain data of physical conditions 

20 of the subject and, therefore, is ensured of knowing the 
subject's conditions with visual images of same. 

According to yet another feature thereof, the invention 
calls for storing data of normal conditions of a subject; and 

25 transmitting the normal condition data together with the 
aforementioned data of his physical conditions to the receiv- 
ing party. With the configuration as above, the receiving 
party is able to assess the physical conditions of the subject 
referring to his normal conditions, thereby more reliably 

30 judging his current condition. 

Furthermore, according to yet another feature of the 
invention, the invention is provided with one or more mobile 
units, each of which has a mobile unit transmitting/receiving 
means for transmitting and receiving vocal signals; a remote 

35 unit which has a remote unit transmitting/receiving means 
for transmitting and receiving vocal signals as well as 
receiving visual signals, the remote unit also having an 
image display means for displaying visual images; one or 
more imaging devices; and a management unit which has an 

40 imaging device controlling means for controlling the imag- 
ing function of the picture-taking devices, and a relay means 
for relaying communication between the aforementioned 
mobile unit transmitting/receiving means and remote unit 
transmitting/receiving means. With the configuration as 

45 above, the remote is capable of receiving signals represent- 
ing visual images of a subject taken by imaging devices, and 
voice communication is possible between the remote unit 
and the mobile unit. 
According to yet another feature of the invention, the 

50 remote unit is provided with picture-taking directing means 
for directing the manner of performance of the imaging 
devices. With the configuration as above, which permits the 
remote unit to easily direct how pictures of a subject are to 
be taken, it is easy for the user of the remote unit to obtain 

55 desired pictures. 

According to yet another feature thereof, the invention is 
provided with a physical condition monitoring means for 
monitoring the physical condition of a subject; and the 
mobile unit transmitting'receiving means of each mobile 

60 unit is capable of transmitting the physical condition data 
through the relay station to the remote unit transmitting/ 
receiving means. With the configuration as above, the user 
of the remote unit is able to easily obtain data concerning the 
physical condition of the subject and, therefore, is ensured of 

65 knowing the subject's condition with visual images of same. 
According to yet another feature thereof, the invention is 
provided with a physical condition storage means for storing 
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data of normal condition of a subject; and a relay means to 
transmit the normal condition data together with the afore- 
mentioned data of his physical condition to the remote unit 
transmitting/receiving means. With the configuration as 
above, the user of the remote unit is able to assess the : 
physical condition of the subject in the context of his normal 
condition, thereby more reliably judging his current condi- 

According to yet another feature thereof, the invention is 
provided with an image composite means for combining i 
visual images of a subject taken by an imaging device with 
data of his physical condition which has been monitored by 
the physical condition monitoring means. With the configu- 
ration as above, it is easy for the user of the remote unit to 
know the status of the subject and data of his physical 1 
condition by way of looking at the pictures. 

According to yet another feature of the invention, each 
mobile unit is provided with an extracting means to extract 
data for its own location; the mobile unit transmitting/ 
receiving means transmits the locational information 2 
extracted by the extracting means to the relay means; the 
management unit is provided with a receiving means to 
receive said locational information from the mobile unit; and 
the imaging device controlling means controls performance 
of the imaging devices based on the locational information 2 
received as above. With the configuration as above, it is easy 
to know the location of the sender and selectively control the 
appropriate imaging devices with the imaging device con- 
trolling means. 

The above, and other objects, features and advantages of " 
the present invention will become apparent from the fol- 
lowing description read in conjunction with the accompa- 
nying drawings, in which like reference numerals designate 
the same elements. 

BRIEF DESCRIPTION OF DRAWINGS 
FIG. 1 is a block diagram illustrating ; 



n embodiment of the present ir 



FIG. 2 is a front v: 
>ystem according tc 



it of the present ii 



FIG. 12 is a block diagram illustrating a 
system according to another embodiment of the present 
invention; and 

FIG. 13 is a block diagram illustrating a communication 
system according to yet another embodiment of the present 



FIG. 3 is an explanatory drawing of disposition of imag- 
ing device of a communication system according to an 
embodiment of the present invention; 

FIG. 4 is an explanatory drawing to show how an imaging 
device placed on a cart, is used; 

FIG. 5 is a block diagram illustrating the extraction of a 
location by means of a plurality of receiving devices of a 
communication system according to an embodiment of the 
present invention; 

FIG. 6 is an explanatory drawing to further illustrate the 
principle triangulation range finding during the location 
extraction illustrated in FIG. 5; 

FIG. 7 is an explanatory drawing to illustrate range 
finding by an imaging device; 

FIG. 8 is an explanatory drawing illustrating a properly 
framed subject; 

FIG. 9 is an explanatory drawing to illustrate the rela- 
tionship between the distance and zooming with said imag- 
ing device; 

FIG. 10 is an explanatory drawing to illustrate the prin- 
ciple of location detection by means of GPS; 

FIG. 11 is an explanatory drawing to illustrate a golf 
course divided into cells; 



Next, a communication system according to an embodi- 
ment of the present invention is explained hereunder, refer- 
ring to the drawings, where the communication system is 
utilized on a golf course. 

Referring to FIG. 1, numeral 1 denotes a mobile unit 
which is in the possession of each player or caddie. Each 
mobile unit 1 is capable of communicating through a wire- 
less means, such as radio waves, with management unit 2, 
which is installed in a clubhouse and serves as a relay 
station. Through ISDN or public lines, such as a general 
telephone line, or wireless means, management unit 2 is 
capable of communicating with a remote unit 3 installed in 
such facilities as a hospital. 

As shown in FIG. 2, mobile unit 1 is provided with a 
liquid crystal display 5 to display guidance and other 
information, a ten-key pad 6 to input scores and so forth, a 
function display key 7 to display various functions to be 
performed, a select/execute key 8 and scroll keys 9, as well 
as a light pen 10 attached to the mobile unit and an 
emergency button 11 located on the topside of the unit. And 
as shown in FIG. 1, mobile unit 1 is provided with a 
transmitter 15 and a voice communication device 16. Trans- 
mitter 15, provided with its own identification number, 
which may be, for example, the personal identification 
number of the player in possession of the mobile unit 1, and 
transmits signals revealing its location together with its 
identification, and as appropriate, notification of an emer- 
gency. Mobile unit 1 is also provided with voice commu- 
nication device 16 having voice communication capability. 
In addition to the above mentioned functions, mobile unit 1 
is also capable of performing such functions as score 
tabulation, detecting its own location, guidance display, 
settling accounts and so forth. 

Referring again to FIG. 1, management unit 2 is provided 
with a receiver 21 which receives signals from transmitter 
15 of mobile unit 1 either directly or by way of antennae 
disposed at appropriate locations, and a controller compo- 
nent 22 which executes control commands based on infor- 
mation received from receiver 21. Further, controller com- 
ponent 22 is connected to transmitter 23 which a 
voice communication device 16 of a mobile unit 1 
addition, controller component 22 is provided with v 
communication device 24 which allows vi ' 
tion with voice communication device 16 of mobile unit 1 to 

; occur after voice communication device 16 is activated. 
Controller component 22 is also connected to a camera 
controller component 26, which issues commands pertaining 
to the selection, direction, angle, and when the situation 
demands, the distance, of each respective imaging devices 

i 27j~27„, 27 c . Further, signals representing visual images 
from any one of imaging devices 27j~27„, 27 c are received 
by visual image receiver component 28, with visual image 
receiver component 28 and voice communication device 24 
being connected to a relay device 25. 

; Remote unit 3 is provided with a voice communication 
device 31, which may be a telephone or the like, and a visual 
image receiver component 32 which may be a monitor. 
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Voice communication device 31 and visual image receiver 
component 32 are connected to relay device 25 of manage- 
ment unit 2 by such means as the aforementioned public 
transmission lines. 

As shown in FIG. 3, imaging devices 27 j to 27„ are placed 5 
around the perimeter of each hole. Further, as shown in FIG. 
4, an on-vehicle imaging device 21 a may be placed atop a 
golf cart 29, in which case on-vehicle receiver 30 which 
receives control signals from camera controller component 
26 is also placed atop a golf cart 29, imaging device 27 ' c 10 
being driven by on-vehicle receiver 30. Each imaging device 
27J-27,,, 27 e is respectively provided with mechanisms to 
perform panning, tilting, as well as zooming, and may also 
be provided with auto-focus capability. 

Selection of imaging devices 27 1 ~27„, 27 c as well as J5 
control of panning, tilting and zooming of the selected 
imaging devices 27,. is performed by imaging device con- 
troller component 26. However, on-vehicle camera 27 c need 
not be among any of said imaging devices 27,-. Further, in 
cases where imaging devices are to be automatically 
selected, an imaging device 27J-27,,, 21 c nearest the subject 
to be imaged, for example, can be selected. 

As shown in FIG. 5, a plurality of relay devices 17J-17,, 
and direction determining components 33j~33„ may be 
provided, relay devices 17J-17,, being respectively provided 
with directional antennae and corresponding to said direc- 
tion determining components 33j~33„ so that the location of 
transmitter 15 may be detected by a location computing 
component 34 based on the direction determined by these 
direction determining components 33j~33„. As to a number 
of relay devices 17 j-17,,, at least two devices are needed to 
determine a direction and a distance by using the principle 
of triangulation. 

Next, referring to FIG. 10, another embodiment of the 
present invention is explained hereunder, said embodiment 
calling for detecting a location by means of mobile unit 1 
itself and transmitting signals including the data for the 
location thus detected. 

In this case, the direction and distance to mobile unit 1 
from any of imaging devices 27j~27„, 27 c can be deter- 
mined by means of, for example, providing each hole with 
a plurality of antennae 41 which transmit reference radio 
waves; or as a second example providing mobile unit 1 with 
a navigation function such as GPS (Global Positioning 
System) which uses reference radio waves from a plurality 
of satellites 42. In these examples, mobile unit 1 converts the 
vectorial information of its location relative to each of 
reference antennae 41 or satellites 42, or a combination 
thereof, into location in terms of coordinates, such as 
latitude and longitude. Consequently, the direction and dis- 
tance to mobile unit 1 from any one of imaging devices 
27!~27„, 27 c can be computed from the absolute coordinate 
locations of mobile unit 1 and the imaging device. 

As shown in FIG. 11, the terrain of a hole or the entire 
course may be divided into cells in a matrix, each cell having 
its own ID number, so that the location of mobile unit 1 may 
be represented by a cell number. In cases where this cell 
system is applied, data of the respective directions and 
distances to the cells with respect to each imaging device 
27-,~27„, 27 c may be stored beforehand so that the appro- 
priate imaging device 27 1 ~27„, 27 c can be selected and 
controlled by directly using said data. Furthermore, the most 
suitable imaging device 27 1 ~27„, 27 c to be used for each cell 
may be included in the cell data to be stored. Cells are 
formed by dividing the entire course of the terrain from the 
tee ground to the green of each hole into small areas in a 
matrix which measures, for example, 3 meters on a side. 
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Operation of the above embodiments is described here- 
in case of an emergency such as when a player is seriously 
injured or collapses, by operation of emergency button 11 of 
mobile unit 1 by the player himself or a person accompa- 
nying him, mobile unit 1 transmits an emergency signal as 
well as a locational signal indicating its own location, as 
determined by the location determining component 
described for FIG. 10, above together with its ID data to 
receiver 21 of management unit 2. 

Alternatively, the system may be configured such that the 
location of mobile unit 1 is computed by location computing 
component 34 shown in FIG. 5 and that signals including the 
locational signal indicating the location computed by this 
location computing component 34 are received by receiver 
21. In this case, the location is determined based on signal 
receiving conditions of the plurality of relay devices 
17j-17„. In other words, as distance 1 between two relay 
devices 17 3 and 17 2 is constant as shown in FIG. 6, the 
location of transmitter 15 that transmitted the signals is 
determined by location computing component 34 by ascer- 
taining the respective directions to transmitter 15 from relay 
devices 17j and 17 2 , and the signals which include the 
locational signal indicating the current location of the spe- 
cific transmitter 15 are transmitted to receiver 21. 

As a result, according to the ID data included in the 
signals received by receiver 21, controller component 22 
controls transmitter 23 to send signals to activate voice 
communication device 16 of mobile unit 1, thereby allowing 
voice communication between voice communication 
devices 16 and 24. A number of communication channels 
during normal circumstances can be reduced by preventing 
voice communication device 16 from being activated under 
normal circumstances and granting voice communication to 
only a specific mobile unit 1 in case of an emergency. 

Based on the locational signal of receiver 21, camera 
controller component 26 of controller component 22 is 
controlled so that imaging device controller component 26 
selects the most suitable camera 21 1 ~21,„ 21 \ according to 
the criteria set beforehand. Panning and tilting of imaging 
device 21 } ~21,„ 21 c selected by camera controller compo- 
nent 26 is controlled in order to adjust the angle of its view. 
Selected imaging device 27 1 ~27„, 21 c determines the 
45 distance to the subject through control by camera controller 
component 26 or by automatic focusing using components 
incorporated in the camera itself, such as light emitting 
elements and photo-electric elements or, as shown in FIG. 7, 
an ultrasonic transmitter 27e and ultrasonic receiver 21r, 
50 with automatic focusing being performed by emitting light 
or signals from the light emitting elements or ultrasonic 
transmitter 21e and computing the distance to the subject 
based on the length of time from when the light or the signals 
are emitted to when they are received by the photo-electric 
55 elements or ultrasonic receiver 21r. 

At this time, imaging device 21 1 ~21 n , 21 c is zoomed so 
that the proportion of the image of the subject in relation to 
the size of the frame is constant as shown in FIG. 8. In other 
words, when the distance from imaging device 21 1 ~21„ or 
60 21 c to the subject is L, the height of the subject M, the focal 
length of the lens of the camera F, and the length of the 
image of the subject in the frame K as shown in FIG. 9, K 
is approximately equal to FxM/L, if value L is sufficiently 
large. Therefore, granting that M is the average height of an 
65 adult and constant, K can be made constant by changing 
focal length F according to distance L to transmitter 15 
which has been measured. 
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The signals representing visual images of the person who 
is injured or ill, these pictures being taken by imaging 
devices 27j~27„, 27 t ., are received by visual image receiver 
component 28. The visual signals and vocal signals, which 
have been received from voice communication device 16 
through voice communication device 24, are transmitted by 
relay device 25 through public lines respectively to visual 
image receiver component 32 and voice communication 
device 31 of remote unit 3. 

Further, where a doctor or other person using remote unit 
3 wants specific pictures, the system may have such a 
configuration as to permit him to operate a dial or buttons or 
the like on voice communication device 31 to transmit 
DTMF signals through relay device 25 to the imaging device 
controller component 26 so that camera 21^21,,, 27 c is 
selected and controlled to perform zooming, panning and 
tilting. This can be done by a setting such that, for example, 
pushing 2 causes upward tilting; pushing 8 downward 
tilting; pushing 4 panning to left; pushing 6 panning to right; 
pushing 1 zooming up; pushing 7 zooming down; and 
pushing * and the ID number of the desired camera when 
changing cameras. 

The configuration as above facilitates looking at visual 
images of the patient on the course in favorable conditions 
by means of remote unit 3 installed in a hospital or the like 
and also provides communication with mobile unit 1 
through telephone or other means which serves as voice 
communication device 31. As the invention thus makes it 
possible to know the conditions of the patient and give 
necessary directions for treatment of the patient and so forth, 
it has a merit in improving the effect of first aid treatment. 

Further, as management unit 2 at a clubhouse can be 
connected to remote unit 3 at a hospital through public lines, 
installation of a system according to the invention is rela- 
tively easy. 

Next, another embodiment is explained hereunder, refer- 
ring to FIG. 12. 

The embodiment shown in FIG. 12 is a simpler version of 
the embodiment shown in FIG. 1 and docs not have receiver 
23 of management unit 2. In this embodiment, communica- 
tion between voice communication device 16 of mobile unit 

1 and voice communication device 24 of management unit 

2 is activated by operating emergency button 11. 
Omitting receiver 23 makes necessary an increased num- 
ber of channels in order to avoid radio interference. On the 
other hand, it is advantageous in providing immediate corn- 
Next, yet another embodiment is explained hereunder, 

referring to FIG. 13. 

The embodiment shown in FIG. 13 has the same configu- 
ration as the embodiment shown in FIG. 12 except that it is 
capable of monitoring various physical conditions. 
Furthermore, a system having the configuration as the 
embodiment shown in FIG. 1 may also have a means of 
monitoring physical conditions. 

According to the configuration of the embodiment, trans- 
mitter 15 of mobile unit 1 is capable of being connected to 
a monitoring instrument 51 for various physical conditions, 
such as body temperature, pulse, blood pressure and so on, 
through a wire circuit or wireless means. 

Said monitoring instrument 51 may be of various types: 
for example, a constantly-worn type to be worn all the time, 
which may be in the shape similar to a wrist watch and 
measures body temperature, blood pressure, pulse and so on 
from the temperature of the surface of the body and blood 
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vessels around the wrist by means of a thermometer, a 
pressure gauge, an infrared sensor or the like; an in-contact- 
when-necessary type to be brought into contact with a part 
of body, such as inside the mouth or an arm, to measure body 

5 temperature, blood pressure, pulse and so on by means of a 
thermometer, a pressure gauge, an infrared sensor or the like 
whenever necessary; and a fixed-installation type installed at 
a fixed location such as at a rest area in a clubhouse or a shop 
on the grounds of the course. Said fixed-installation type 

10 may have the configuration which calls for inputting the ID 
date of the user through his mobile unit 1 or other means, 
and setting a part of the user's body, such as an arm, on the 
equipment to obtain information of his physical condition. 
In cases where the constantly-worn type or the in-contact- 

15 when-necessary type is used, data of body conditions is 
detected near the user's mobile unit 1. Therefore, the data 
may be input into mobile unit 1 by connecting it to moni- 
toring instrument 51 or, because they are at close range, 
through weak radio waves. In case of the fixed-installation 

20 type, data may be directly input to management unit 2 
without going through mobile unit 1. 

Furthermore, mobile unit 1 is provided with a receiver 52 
for receiving signals which inform of abnormal physical 
conditions. Upon receiving such warning signals from man- 

25 agement unit 2, receiver 52 gives warning by voice or 
sounding alarm, as well as displaying a message, such as 
"Body temperature is rising, presenting danger of heat 
stroke: Drink fluids and rest in a cool place", "Blood 
pressure above normal range: Stop game and return to 

30 clubhouse", "Irregular pulse: Stop game temporarily and 
rest at the nearest rest area for at least 30 minutes" or the like 
on liquid crystal display 5. 

Management unit 2 is provided with a physical condition 
storage component 53 for storing data of physical condition 

35 monitored by monitoring instrument 51 and transmitted 
from mobile unit 1, physical condition storage component 
53 storing transmitted data in the order of elapsed time from 
prior to starting the game and thereafter. Physical condition 
storage component 53 is also connected to relay device 25. 

Physical condition storage component 53 is also con- 
nected to ajudging component 54 which judges whether 
abnormal physical conditions are being experienced by 
comparing current conditions with those stored in physical 
condition storage component 53. Further, judging compo- 

45 nent 54 is connected to a transmitter 55, which transmits 
warning signals to receiver 52 of mobile unit 1 when 
physical conditions of the subject is judged to be abnormal. 
Remote unit 3 is provided with a physical condition 

5Q display component 55 which has image composite function 
and is capable of generating composite images representing 
physical conditions of a patient on the display. 

Next, operation of the above embodiments shown in FIG. 
13 is described hereunder. 

55 Its operational procedure is basically the same as that of 
the embodiment shown in FIG. 12. 

First of all, store data of normal conditions of each player 
in physical condition storage component 53 beforehand by 
inputting it using keyboard 6 or assessing his physical 

60 conditions prior to starting the game by means of monitoring 

Then, during the game, check physical condition at regu- 
lar intervals or whenever desired by means of monitoring 
instrument 51. The data of this assessment is stored in 
65 physical condition storage component 53 in the order of 
elapsed time and also compared by judging component 54 
with the data of the normal condition or the data of the 
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condition before starling the game, which is also stored in 
physical condition storage component 53. In cases where 
any abnormal condition is delected in this judgement step, 
judging component 54 causes transmitter 55 to send out 
warning signals. When the warning signals have been 5 
transmitted, receiver 52 sounds an alarm and displays an 
appropriate message, such as those described above, on 
liquid crystal display 5. 

When emergency button 11 is activated, in addition to 
sound signals and visual signals of pictures taken by one of 10 
imaging devices 21^-21,,, 27 c , data of physical conditions of 
the subject stored in physical condition storage component 
53 is transmitted through relay device 25 to remote unit 3. 

In remote unit 3, the physical condition data stored in 
physical condition storage component 53 is combined with 
pictures taken by imaging device 27 1 ~27„, 27 c and dis- 
played in the form of composite image. Physical condition 
data to be displayed includes data of normal condition, 
physical condition before starting the game and those 
obtained by assessment after starting the game, as well as the 20 
current condition. As these physical conditions are displayed 
in the order of elapsed time, with the changes of physical 
condition and the current state being thus presented, the 
changes in the condition of the patient can be reliably ^ 
known. 

Furthermore, the patient's chronic illness, allergies, cur- 
rent medication and other data may be stored in physical 
condition storage component 53 beforehand using ten-key 
pad 6 of mobile unit 1 or other means. 30 

As it is possible to know about the physical condition of 
the person who has been injured or fallen ill, following the 
flow of time and in real time according to this embodiment 
shown in FIG. 13, the embodiment ensures of giving appro- 
priate care in an emergency and effective early-stage treat- 35 

Furthermore, the term "sender" referred to in the above 
explanation and appended claims, whose pictures are being 
taken, is not limited to the person who actually sends 
signals; it may also mean the injured or the sick person, who 40 
is at a location near the actual sender and whose physical 
conditions are monitored. 

Having described preferred embodiments of the invention 
with reference to the accompanying drawings, it is to be 
understood that the invention is not limited to those precise 45 
embodiments, and that various changes and modifications 
may be effected therein by one skilled in the art without 
departing from the scope or spirit of the invention as defined 
in the appended claims. 

1. A communication method comprising the steps of: 
transmitting signals from a sender by means of a mobile 

unit carried by said sender, said signals having a vocal 

and an informational component; 
receiving said signals from said sender at a relay station; 55 
selecting a selected imaging device from a plurality of 

imaging devices located at various positions of an area; 
obtaining a visual image of said sender, said obtaining a 

visual image including controlling a plurality of imag- 60 

ing functions of said selected imaging device; 
transmitting further signals from said sender together with 

said visual image from said relay station to a receiving 

party; and 

transmitting vocal signals of said receiving party through 65 
said relay station to said sender responsively to at least 
one of said further signals and said visual image. 
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2. The communication method of claim 1, wherein: 
said informational component includes locational data for 

said sender's location; 
said selecting a selected imaging device is responsive to 

said locational data; and 
said controlling said plurality of imaging functions of said 

imaging device is responsive to said locational data. 

3. The communication method of claim 2, wherein: 
said selecting a selected imaging device includes one of 

automatically selecting at said relay station and manu- 
ally selecting by said receiving party; and 
said controlling said plurality of imaging functions 
includes one of automatically controlling at said relay 
station and manually controlling by said receiving 
party. 

4. The communication method of claim 3, wherein said 
transmitting said informational component of said signals 
from said sender includes transmitting a physical condition 
datum of a subject. 

5. The communication method of claim 4, wherein said 
transmitting a physical condition datum includes the steps 
of: 

monitoring a physical condition of said subject to produce 

a present physical condition datum; 
providing a stored normal physical condition datum of 

said subject; and 
correlatively transmitting said stored normal physical 

condition datum and said present physical condition 

datum. 

6. A communication system comprising: 
at least one imaging device; 

at least one mobile unit having means for transmitting and 
receiving vocal and informational signals; 

at least one remote unit having means for transmitting and 
receiving said vocal and informational signals; 

said at least one remote unit having means for receiving 

said at least one remote unit having means for displaying 
visual images responsively to said visual signals; 

a management unit having means for individually con- 
trolling imaging functions of each of said at least one 
imaging device; 

said management unit having a means for relaying said 
vocal, said visual, and said informational signals 
between each said at least one mobile unit means for 
transmitting and receiving and each said at least one 
remote unit means for transmitting and receiving; and 

said management unit having means for concurrently 
receiving said visual signals from said at least one 
imaging device with said signals from said at least one 
mobile unit, and concurrently transmitting said visual 
signals from said at least one imaging device with said 
signals from said at least one mobile unit to said at least 

7. The communication system of claim 6, wherein said at 
least one remote unit has an override means for individually 
controlling imaging functions of each of said at least one 
imaging device. 

8. The communication system of claim 7, further com- 
prising: 

means for monitoring a physical condition of a subject, 
said means for monitoring having physical condition 
output data; 

means for sending said physical condition output data to 
said mobile unit; and 
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said informational signals including said physical condi- 
tion output data. 

9. The communication system of claim 8, further corn- 
means for storing data of a normal physical condition of 

said subject; and 
means for combining said normal physical condition data 
of said subject and said physical condition output data 
of said subject to form composite physical condition 

said informational signals including said composite 
physical condition data. 

10. The communication system of claim 9, further corn- 
means for combining visual images of said subject taken 

by said at least one imaging device with said output 
data of said subject's means for monitoring physical 

11. The communication system of claim 6, further com- 
prising: 

means for locating said at least one mobile unit; means for 
extracting locational data from said means for locating; 

means for sending said locational data extracted by said 
means for extracting to said management unit; 

said management unit having means for selecting said at 
least one imaging device responsive to said locational 
information received from said means for locating; and 

said means for individually controlling controls said at 
least one imaging device selected by said means for 
selecting responsively to said locational information 
received from said means for locating. 

12. The communication system of claim 7, further com- 
prising: 

means for locating said at least one mobile unit; 
means for extracting locational data from said means for 
locating; 

means for sending said locational data extracted by said 
means for extracting to said management unit; 

said management unit having means for selecting said at 
least one imaging device responsive to said locational 
information received from said means for locating; and 

said means for individually controlling controls said at 
least one imaging device selected by said means for 
selecting responsively to said locational information 
received from said means for locating. 
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13. The communication system of claim 8, further corn- 
means for locating said at least one mobile unit; 
means for extracting locational data from said means for 

locating; 

means for sending said locational data extracted by said 

means for extracting to said management unit; 
said management unit having means for selecting said at 
10 least one imaging device responsive to said locational 
information received from said means for locating; and 
said means for individually controlling controls said at 
least one imaging device selected by said means for 
15 selecting responsively to said locational information 
received from said means for locating. 

14. The communication system of claim 9, further com- 
prising: 

means for locating said at least one mobile unit; 
20 means for extracting locational data from said means for 
locating; 

means for sending said locational data extracted by said 

means for extracting to said management unit; 
said management unit having means for selecting said at 
25 least one imaging device responsive to said locational 
information received from said means for locating; and 
said means for individually controlling controls said at 
least one imaging device selected by said means for 
3Q selecting responsively to said locational information 
received from said means for locating. 

15. The communication system of claim 10, further com- 
prising: 

means for locating said at least one mobile unit; 
35 means for extracting locational data from said means for 
locating; 

means for sending said locational data extracted by said 

means for extracting to said management unit; 
said management unit having means for selecting said at 
40 least one imaging device responsive to said locational 
information received from said means for locating; and 
said means for individually controlling controls said at 
least one imaging device selected by said means for 
4J selecting responsively to said locational information 
received from said means for locating. 
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In a method for manipulation of a movable object displayed 
in a television picture, the distance between the object and 
fixed basic positions is detected at a time t together with the 
distance between the object and a television camera in a 
known position. The object's position is converted to a 
position X,Y in the television camera's picture plane, gen- 
erating therein a synthetic object which overlays the mov- 
able object and represents it in the television picture. In a 
method for generating a synthetic track which represents the 
path of a movable object displayed in television pictures 
during a period 6, the path of the object is calculated on the 
basis of its detected positions, and these positions are used 
for generating a synthetic track which is displayed in a 
television picture in order to represent the path of the object 
in the period T. 
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Fig. 3d 
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Fig. 9 a 
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METHOD AND SYSTEM FOR 
MANIPULATION OF OBJECTS IN A 
TELEVISION PICTURE 

BACKGROUND OF THE INVENTION 

The invention concerns a method for manipulation of at 
least one movable, natural object in a natural television 
picture, wherein the television picture is generated by one or 
more television cameras. The invention also concerns a 
method for generating at least one synthetic track in a 
television picture, wherein the synthetic track represents the 
path of a movable natural object in a natural television 
picture during a given period 6, and wherein there is 
employed a method for manipulation of the movable, natural 
object in the television picture. Finally, the invention con- 
cerns a system for implementation of the method for 
manipulating at least one movable, natural object in a natural 
television picture, wherein the television picture is generated 
by one or more television cameras, together with implemen- 
tation of the method for generating a synthetic track in a 
television picture, wherein the synthetic track represents the 
path of a movable, natural object in the television picture 
during a given period 9, and wherein a method is employed 
for manipulating the movable, natural object in the televi- 

In television broadcasts where a movable, natural object 
plays a central part in the broadcast, it can often be difficult 
to follow the object or the movement thereof in the televi- 
sion picture. This is the case, e.g., in sports broadcasts from 
various ball games, such as football, handball, tennis, golf 
and ice hockey, where the picture format used, the back- 
ground of the picture, colour, light conditions etc. can make 
it difficult to follow the object or the movement thereof. The 
object may also be invisible for a brief or a considerable 
period because it is masked by other objects in the picture. 
Such problems can be a factor in influencing the popularity 
of the television broadcasts, which in turn can have conse- 
quences for the sponsors' choice of programs or purchase of 
commercials in such programs, since the viewing figures are 
not as high as is desirable. 

In order to make a game like ice hockey more attractive 
to the television medium, on the basis of the usual complaint 
about television coverage of ice hockey matches that the 
puck is difficult to follow, it has been proposed to make the 
ice hockey puck more clearly visible in the television picture 
by employing a special puck which is equipped with a 
number of infrared emitters driven by a battery provided in 
the puck. On the rink the puck is tracked by sensors placed 
along the edge of the rink, the sensors transmitting infor- 
mation to a camera which is connected to a computer. The 
information is fed to a data processing centre where a signal 
is generated which is processed graphically and introduced 
into the television picture. The puck can thereby be caused 
to change shape or colour. This system has been introduced 
by the company Fox Sports in the USA and is employed to 
represent the puck, e.g., surrounded by a shining halo in 
some colour or other and equipped with a coloured "comet 
tail" when the puck is in motion, e.g. at a certain speed. The 
colours can, however, be altered according to the wishes of 
the producer. In a practical embodiment of this system which 
is called "FoxTrax", 16 sensors are employed around the 
rink and two infrared cameras which also follow the puck. 
The processed representation of the puck is introduced into 
the television pictures which are recorded by the normal 
television cameras, and the special colour effects such as the 
halo and the "comet tail" are overlaid the television signal 
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with a delay which is not greater than between Vis and Vs of 
a second. "FoxTrax" is described in U.S. Pat. No. 5,564,698. 

However, the system is complicated and offers limited 
opportunities for manipulation of the object, i.e. the picture 

5 of the puck. Another drawback is that the infrared emitters 
in the puck are driven by a battery which has a limited life, 
with a life of only 10 minutes having been quoted. This, 
however, is sufficient to allow the puck to be replaced during 
change of sides in a period of an ice hockey match, but 

10 makes a similar system less suitable in games where the 
object, i.e. the puck or ball, is in play for a longer period. 

U.S. Pat. No. 4,675,816 (Brendon & Vinger) discloses a 
method for electronic localization of the ball in American 
football. The object of the method is to determine whether 

j 5 the ball has moved 10 yards forwards and provides simul- 
taneous precise localization information about the ball and 
the possibility of positioning the football on the field. This 
method is regarded as an aid for spectators, officials and 
television. It comprises steps for providing a radio transmit- 

2Q ter in the football and transferring radio signals from the 
football to a number of rotating receiver antennas, the 
antenna being directed towards the radio transmitter in such 
a manner that the radio waves supply accurate angular 
direction signals which can be used to calculate the position 

25 of the radio transmitter and the football on the field. These 
angular direction signals are supplied to a microprocessor 
which calculates the position of the transmitter and the 
football, with the use of a triangulation method which 
includes a computation stage which also indicates how far 

30 the ball has moved forwards, and the results of this com- 
putation stage can be shown on display units with a view to 
spectators and television viewers as well as being transferred 
to a control unit which is employed by the officials in order 
to monitor the course of the game. 

35 From U.S. Pat. No. 5,138,322 (Nuttel) there is also known 
a system for continuous and precise measurement of the 
positions of a generally symmetrical object, such as a tennis 
ball, which is in motion in a predetermined three- 
dimensional area, such as a tennis court. In this case a 

40 number of antennas are employed which transmit radar 
signals to the three-dimensional area, reflected return signals 
from the ball being detected and compared with the trans- 
mitted signals for phase determination of the return signals, 
thus enabling unambiguous distances to the object or the ball 

45 to be determined. For this purpose a statistical method is 
employed in order to achieve an accurate determination of 
distance. The path of the object or the ball can be calculated 
simultaneously, and the system is calibrated by placing 
signal reflectors in different fixed positions on the court. A 

50 Doppler radar technique is employed. 

Furthermore, in U.S. Pat. No. 5,346,210 (Utke & al.) there 
is disclosed an object localisation system, especially for 
localizing the ball in a special playing situation in American 
football. The system employs three sensors placed on one 

55 side of the field and a calibration source placed on the other 
side. The calibration source transmits an ultrasound signal 
which is received by the sensors and a ball marking unit 
which can be placed on the field instead of the ball also 
transmits an ultrasound signal which is received by the 

60 sensors, together with an RF signal which is received by the 
calibration source in order to switch it off. The sensors emit 
signals which are used by a processing unit to calculate time 
delays in order to determine the ball's position. 
Alternatively, the ball marking unit may only transmit an RF 

65 signal which is received by the sensors which in turn emit 
signals which are processed in order to determine time 
delays between the receipt of the signals in the sensors. In 
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ic ball marking unit which is moved on 
a track is controlled by the processing unit in order to create 
"cally a visual representation of the position of the 



None of the above-mentioned, known systems, however, 
is particularly well suited to achieve full freedom to manipu- 
late the picture of a natural object in a television picture, all 
requiring the use of relatively expensive and complicated 
systems. Nor are they suitable for all types of games or 
sports and moreover they appear to be substantially '■ 
restricted to use in sporting events, but other forms of 
television broadcasts are, however, conceivable, where it 
will be equally interesting to be better able to visualise a 
movable object in a television picture with suitable detection 
and processing methods. 

SUMMARY OF THE INVENTION 

Consequently, it is a first object of the present invention 
to provide a system for manipulating the picture of at least 
one movable, natural object in a natural television picture in : 
such a manner that the object's position and movement are 
clearly visible in the television picture. 

A second object of the present invention is to provide a 
synthetic representation of the natural object, thus enabling 
the synthetic representation to appear as a synthetic object in 
the television picture and to represent the natural object's 
movement and position. 

A third object of the present invention is to manipulate the 
synthetic object in the television picture with regard to shape , 
and colour in such a manner that the viewer will have no 
trouble in following the object. 

A fourth object of the present invention is to determine the 
path of a movable, natural object and to visualise this path 
in the form of a synthetic track for the object in the television 

Finally, it is an object of the present invention to provide 
a system which makes it possible to detect the movement of 
such a natural object and to process the detected data in 
order to generate and manipulate a representation of the , 
movable object in a television picture as well as generating 
a synthetic track which represents the movement of the 
natural object in the television picture. 

The above-mentioned objects and other advantages are 
achieved with a method which according to the invention is • 
characterized by detecting the distance between the object 
and at least 2 fixed basic positions in a preselected x,y,z 
co-ordinate system at a time t, each basic position corre- 
sponding to a known position of a detector, determining an 
x,y,z co-ordinate for the object in the preselected x,y,z : 
co-ordinate system at the time t, determining the distance 
between the camera's lens centre and the object at the time 
t as an object vector in the preselected co-ordinate system, 
determining the television camera's optical axis in the 
preselected co-ordinate system at the time t as a camera : 
vector in the preselected co-ordinate system, determining a 
line from the television camera's lens centre to the point of 
intersection between the edge of the generated television 
picture and the plane formed between the object vector and 
the camera vector at the time t as a zoom vector, the object 
vector being located between the camera vector and the 
zoom vector when the object is visible in the television 
picture at the time t, and when the object is not visible in the 
television picture at the time t the zoom vector is located 
between the camera vector and the object vector, and deter- 
mining an X,Y position of the object referred to the televi- 
sion camera's picture plane and the camera vector on the 



basis of the object vector and the camera vector at the time 
t, and if the object vector is located between the camera 
vector and the zoom vector, to insert a synthetic object in the 
X,Y position in the television picture at the time t, the 
synthetic object constituting a representation of the natural 
object recorded by the camera at the time t, or if the zoom 
vector is located between the camera vector and the object 
vector at the time t, to insert a symbol in the television 
picture, the symbol indicating the location of the X,Y 
position of the natural object outside the edge of the picture 
referred to the television camera's picture plane and the 

The above-mentioned objects and advantages are further 
achieved with a method which according to the invention is 
characterized by calculating the path of the natural object on 
the basis of detected positions x,y,z for the natural object in 
a preselected x,y,z co-ordinate system at the time t, where 
te6, converting the detected positions at the time t to an X,Y 
position in the television camera's picture plane at the time 
t, and generating the synthetic track in the television picture 
as the connecting line between all X,Y positions for the 
natural object in the picture planes of the natural television 
pictures which are recorded sequentially during the period 6. 

The above-mentioned objects and advantages are further 
achieved by implementing the stated methods according to 
the invention with a system which according to the invention 
is characterized in that it comprises a transponder provided 
in the natural object and arranged to react to an optical, 
acoustic or electromagnetic signal received by the transpon- 
der with transmission of a response signal, at least one 
position module with at least 2 position detectors for trans- 
mitting optical, acoustic or electromagnetic signals and 
receiving response signals from the transponder and pro- 
vided in respective basic positions in a preselected x,y,z 
co-ordinate system, together with a signal processor 1 
arranged to determine the distance between a position detec- 
tor and the object at a time t, a computing module connected 
with the signal processor and arranged to compute the x,y,z 
co-ordinates for the object in the preselected co-ordinate 
system at the time t and on the basis of the computed x,y,z 
co-ordinates for a number of times t to calculate a path for 
the object, together with an object vector given by the 
distance between the camera's lens centre and the object at 
the time t, a camera vector given by the camera's optical axis 
at the time t and a zoom vector between the camera's lens 
centre and the point of intersection between the edge of the 
picture and the plane formed between the object vector and 
the camera vector at the time t, the object vector being cither 
located between the camera vector and the zoom vector or 
the zoom vector between the camera vector and the object 
vector, a camera control system connected with the com- 
puting module and arranged to detect or generate values for 
the camera settings, and a manipulator module connected 
with the camera control system and the computing module 
and arranged to a) create a synthetic object in an X,Y 
position for the natural object in the recorded television 
picture at the time t, the synthetic object constituting a 
representation of the natural object recorded by the camera 
at the time t or b) create a symbol in the television picture, 
the symbol indicating the X,Y position for the natural object 
outside the edge of the picture at the time t, c) generate and 
select attributes for the synthetic object, or d) generate at the 
time t a synthetic track in a recorded or generated television 
picture, the synthetic track representing the path for the 
natural object during a period 8 before or up to the time t. 
BRIEF DESCRIPTION OF THE DRAWINGS 

now be discussed in more detail in 
i with embodiments and with reference to the 
accompanying drawing, in which: 
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FIG. 1 illustrates a design of the system according to the 
present invention, e.g. in connection with a ground for a ball 

FIG. 2 is a block diagram of components in a device for 
processing the detected signals in the system according to : 
the present invention, 

FIGS. 3a, b, c, d are a representation of the object in the 
picture plane of a television camera, 

FIGS. 4a, b, c, d are a representation of the object in the 3 
picture plane of a second television camera, where the object 
is partially located in positions outside the picture plane, 

FIG. 5 is a block diagram of a transponder employed in 
the system according to the present invention and based on 
microwave technique, ] 

FIG. 6 is an active transponder for use with the invention 
and based on an acoustic surface wave component, 

FIG. la is a polling signal emitted in the form of pulses 
from a position detector, 

FIG. lb is a response signal emitted in the form of a pulse 5 
sequence from the acoustic surface wave component, 

FIG. 8 is a system according to the present invention 
implemented for detection of several objects which are in 
motion in separate or connected areas, 

FIGS. 9a, b, c illustrate generation of a synthetic track by 
means of the design according to FIG. 1, 

FIGS. 10a, b illustrate generation of a synthetic track by 
means of the design according to FIG. 8, and 

FIGS. 11a, b illustrate generation of synthetic tracks for ; 
two objects by means of the design according to FIG. 8. 



FIG. 1 illustrates the present invention realised at a sports 35 
centre, such as a ground for ball games or an ice hockey rink 
A, where the area A is the projection in an x,y plane of the 
area which has to be covered by position detectors D 1; . . . 
D 4 . The detectors are connected via a signal line L to a data 
processing unit Q. The data processing unit Q is further 40 
connected via a local data bus to a regulator module R,, 
R 2 , R 3 which is assigned to respective television cameras 
Kl, Kz, K 3 and comprises not shown bodies, preferably in 
the form of servos, for setting the cameras together with not 
shown sensors for detection of the camera settings, includ- 45 
ing camera angles and zoom angles. The position detectors 
Dj, . . . D 4 comprise as shown antennas H, and these are 
directive, with their main lobes m^ substantially covering 
the area A and a certain height above the plane of the area 
A. In FIG. 1 the main lobe for the position detector Dj is 50 
indicated by broken lines M^. Both the position detectors 
D 1; . . . D 4 and the television cameras K 1; Kj, K 3 are 
positioned in a preselected x,y,z co-ordinate system, since 
the plane of the rink A as mentioned may lie in the 
co-ordinate system's x,y plane, the z axis then being per- 55 
pendicular to the x,y plane (orthogonal co-ordinate system). 
The positions of both the position detectors D 1; . . . D 4 and 
the cameras K 1; K 2 , K 3 are precisely defined in the x,y,z 
co-ordinate system. In the area A, i.e. on the rink or over it 
at a certain height there is located a movable object N which 60 
is illustrated in FIG. 1 in two positions x,y,z and x',y',z' at the 
times t and t' respectively. In other words, during a time 
interval 6=t'-t the movable object has moved from the 
position x,y,z to the position x',y',z'. The cameras Kj, K^, K 3 
are directed towards the area A and are set so as to cover a 65 
picture field of a certain size, in FIG. 1 for each camera 
indicated by broken lines. The picture axis or the optical axis 



in a camera K is represented by a vector called the camera 
vector V A ., e.g. for camera K 2 , V A - 2 . The camera vector V A - 
passes through the camera's lens centre and the picture 
field's geometrical centre axis. The cameras K ; , IC,, K 3 
which are used naturally have zoom lenses, and the picture 
fields can thereby vary in size according to how the cameras 
K 1; K 2 , K 3 are zoomed. Along the edge of the rink tran- 
sponders T K are provided in fixed positions for calibration of 
the position detectors D 1; . . . D 4 . Corresponding transpon- 
ders may furthermore also be provided both in the position 
detectors D„ . . . D 4 and the cameras K 1; K^, K 3 . 

The connecting line between the lens centre in a camera 
K and the object N which may be a ball or puck or any other 
movable object, is called the object vector and designated by 
V„, y NK2 , e.g., being the object vector for the camera K 2 at 
the time \ and V' jVA - 2 the object vector for the camera IC, at 
the time t'. The same applies to the other cameras. The 
distance from a position detector D to the object N at the 
time t is designated as a and at the time t' as a', for example 
for the position detector T) 1 as a 1 and i\ respectively. 

When a polling signal is emitted from a position detector 
D, after a certain period x a response signal is received from 
the natural object N. The response signal may be a reflection 
of the polling pulse or it may be a response pulse from the 
object N, the response pulse being triggered by the polling 
signal from the position detector D. The time between the 
transmission of the polling signal and receipt of the reply 
signal, i.e. x, now becomes a measurement of the distance a 
between a position detector D and the object N. For 
example, the position detector D 1 finds that the distance to 
N at the time t is a 1; and at the time t 1 n\. The same applies 
to the remaining position detectors. If the object N is now 
located all the time on the x,y plane in the area A, its position 
will be unambiguously determined by using two position 
detectors, e.g. Dj and D 2 and finding respective distances 
■d\; a 2 , a' 2 . If the area A is three-dimensional, i.e. the position 
to the object N is determined by the co-ordinates x,y,z at the 
time t, it is necessary to have at least three position detectors, 
for example Dj, D 2 , D 3 . As illustrated in FIG. 1 four position 
detectors D,, . . . D 4 are preferably used to achieve unam- 
biguous detection of the position of the natural object N. The 
position of the object N is thereby solely determined by 
distance measurements, i.e. by trilateration, a method which 
is well known to those skilled in the art and therefore will not 
be discussed further here. Thus, as distinct from distance 
determination by triangulation, it is not necessary to deter- 
mine direction angles to the object N. Moreover, in the 
actual position determination a statistical optimization may 
advantageously be employed in order to reduce any posi- 
tioning errors, such as estimation by means of the least 
squares method. 

As illustrated in FIG. 2, according to the invention the 
system comprises a transponder T provided in the natural 
object N. The transponder T may be a passive transponder, 
such as a reflector for microwave signals or radar signals 
transmitted by the antenna H in the position detector D or it 
may be an active transponder which is triggered by the 
polling signal transmitted by the position detector D and 
emits a response signal which is detected by the position 
detector D. The position detector D with antenna H forms 
part of a position module M which comprises a signal 
processor 1, the signal processor 1 preferably being pro- 
vided in a data processing unit Q. The signal processor 1 is 
connected via the signal line L with, for example, two or 
more position detectors D„ . . . , finding the detection 
distance a on the basis of the measured running times x. For 
this purpose there will be provided in the signal processor 1 
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components which are well known to those skilled in the art, 
including a clock with a very high clock rate, the clock rate 
at least being adapted to the frequency for the polling signals 
emitted from the position detectors D. The detected dis- 
tances a are given to a computing module 2 which by means 
of trilateration calculates the positions x,y,z at different 
times t and thereby also the path of the object N on the basis 
of positions detected at the different times t. The computed 
positions are supplied to a camera control system 3 in the 
data processing unit Q. In the camera control system 3 
values are calculated for the settings of the camera K by 
means of a regulator module assigned to the camera K, and 
the camera settings can be controlled automatically by 
means of a control loop provided in the camera control 
system 3, on the basis of the existing camera settings and 
detected positions for the object N. The optical system 
(objectives and other lenses) in the camera K are indicated 
schematically and designated by Q. 

The regulator module R comprises not shown servos for 
generating the camera settings and not shown sensors for 
detection and recording of the camera settings. One of the 
sensors is an angle sensor for determining the camera angle, 
and use is preferably made of a triaxial angle sensor which 
thereby indicates the direction of the camera's optical axis or 
the camera vector V^,. The angle sensor can be calibrated by 
measuring the angle of direction at a fixed point. .Such angle 
sensors are well known to those skilled in the art and 
therefore do not require further mention here. The camera K 
can naturally also be manoeuvred manually in order to set 
zoom angles and camera angles. 

The data processing unit Q also comprises a manipulator 
module 4 for generating a synthetic object S which corre- 
sponds to the natural object N, the fixed position of the 
natural object being converted and scaled to a corresponding 
(projected) position X,Y in the picture plane of the camera 
K, taking into account the recorded values for the camera 
settings. Thus in the position X,Y in the camera's picture 
plane, via, e.g., a video generator (not shown) provided in 
the manipulator module 4, a synthetic object S can be 
created, representing the natural object N in its position X,Y 
in the picture plane in the camera K, and the synthetic object 
S can be represented with various attributes for size, shape 
and colour. The primary object here is that the synthetic 
object S which represents the natural object N should at all 
times display the position and/or the movement of the 
natural object N, as it would be represented in the television 
picture at any time t. 

The signal processor 1, the computing module 2, the 
camera control system 3 and the manipulator module are all 
interconnected via the local data bus B 1( which also passes 
signals to and from the camera's regulator module R. If a 
plurality of television cameras K are assigned to the data 
processing unit Q, they are also naturally connected to the 
local data bus B x . From the data processing unit Q a global 
data bus B a leads to a production unit P which takes care of 
the actual production of the television broadcast and may 
transfer the pictures to the television station or the television 
transmitter. 

In the production unit P there are displayed the natural 
television pictures recorded at any time, or possibly syn- 
thetically generated television pictures with a synthetic 
object S and the recorded television pictures with the 
synthetic object overlaid in the correct position, thus 
enabling the producer to select the camera and the picture 
which is desired to be represented at any time during a 
television broadcast and transmit the pictures via a standard 
TV line to the television transmitter. If only a data process- 
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ing unit Q is employed, the production unit P and the data 
processing unit Q may preferably be integrated into one unit, 
but if a plurality of data processing units are provided, the 
system may be completely decentralised, with all the data 

5 processing units thus being connected to the production unit 
via the global data bus B G . 

The representation of the synthetic object S in a television 
picture will now be described in more detail with reference 
to FIGS. 3 and 4. As illustrated in FIGS. 3a and 36, camera 

io K 2 , for example, records in a known position and with a 
known camera angle and zoom angle at time t a television 
picture in the picture plane l K2 . The camera vector V K2 is 
normally located on the picture plane \ K2 in the centre 
thereof. The natural object N which at the time t is located 

15 in the x,y,z position has the object vector V NK 2 which forms 
an angle with the camera vector V K2 and passes through the 
lens centre w. As can be seen in FIG. 3b, the two vectors 
and Vjy^ form a plane which intersects the edge of the 
picture plane \ K2 at the point Z 2 . The connecting line 

20 between the lens centre w and the point Z 2 is called the zoom 
vector V Z2 , as illustrated in FIG. 3b, and is thereby deter- 
mined by the camera settings. The representations of the 
remaining vectors in the optical system Q are marked with 
a star, for example V* K2 for the camera vector and V*^, for 

25 the object vector. The zoom vector V Z2 also forms an angle 
with the camera vector V'^ as illustrated in FIG. 3b. If the 
angle between the zoom vector V Z2 and the camera vector 
v *jt2 is greater than the angle between the object vector 
V* NK2 and the camera vector V*^, the natural object is 

30 reproduced in the picture plane I A , 2 and in FIG. 3a illustrated 
represented by a star-like object which constitutes the syn- 
thetic object S. In FIG. 3c the natural object N has moved to 
the position x',y',z' at the time t 1 and the vector parameters 
for the dynamic vectors, i.e. the object vector V N and the 

35 zoom vector V z has changed, so that they are illustrated in 
FIG. 3c as V NK2 and V' Z2 respectively. The same applies to 
FIG. 3d which shows the representation of the vectors in the 
optical system and is marked in the same way as in FIG. 3b. 
The natural object N is still in the picture field and can be 

40 represented by the synthetic object S. In FIG. 4 the picture 
plane \ K3 is shown at two different times t and t' for the 
camera It,. At the time t the position x,y,z of the natural 
object falls outside the camera's picture field. As illustrated 
in FIG. 3b the angle between the camera vector V*^- and the 

45 object vector V* VK3 is greater than the angle between the 
zoom vector and the camera vector V* K3 . In other 
words the object is outside the edge of the picture. This may 
be advantageously indicated in the television picture by 
placing the synthetic object S or an indicator therefor at the 

50 edge of the picture, i.e. at the point Z 3 , the line between the 
centre of the picture and the synthetic object indicating the 
direction of the synthetic object. Naturally, a synthetic 
direction indicator can also be employed to point to the 
object's position outside the edge of the picture. In FIG. 4b 

55 which illustrates the picture plane l' K3 at the time t 1 the 
movable or natural object N has moved into the picture field 
and can be represented by the synthetic object S as shown, 
the angle between the object vector V* NK3 and the camera 
vector V'*, f3 now being smaller than the angle between the 

60 zoom vector V Z3 and the camera vector V'*,^, as can be seen 
in FIG. 4d. 

As mentioned, the attributes of the synthetic object S may 
be freely chosen and the object naturally does not need to be 
a true representation of the natural object, either with regard 
65 to size, shape or colour. An ice hockey puck, e.g., may be 
generated as a highly luminous, pulsing or blinking object in 
a contrasting colour. Moreover, the size or colour of the 
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object can be caused to vary in order to indicate to the viewer 
the apparent distance between him and the natural object, as 
it appears when viewing a television picture. Furthermore, 
the synthetic object can be provided with labels or 
indicators, e.g. of an alphanumerical nature or in the form of 
shining or blinking arrows or other visual indicators in order 
to indicate the distance of the object from, e.g., the camera, 
or also the speed, direction and position of the object. All of 
this can be introduced into the shown television picture in 
connection with the synthetic object S and overlaid the 
natural background in the television picture. The actual 
generation of the synthetic object and other indicators is 
performed in the manipulator module shown in FIG. 2, and 
may, e.g., take place by means of a not shown video 
generator of a type which is well known to those skilled in 
the art. 

The detection of the natural object and the distance 
determination will now be described in more detail. 

There are known and described in the art a number of 
different systems for measuring the distance to an object, 
e.g. by transmitting from an antenna optical, acoustic or 
electromagnetic signals which are reflected back to the 
antenna by the object whose distance has to be determined. 
The measured time difference between the transmission and 
receipt of the signals is a direct measure of the distance to 
the object, since the propagation, velocity of the signals in 
the surrounding medium is assumed to be known with 
sufficient accuracy, and a time measuring system is 
employed which provides the desired accuracy in determi- 
nation of the distance. For example, in U.S. Pat. No. 
3,503,680 (Schenkerman) there is disclosed a distance mea- 
surement system based on a pulse radar. The technique 
therein disclosed takes into account the fact that the object 
whose distance has to be measured can move at a relatively 
great speed, with the result that instead of transmitting a 
pulse and then measuring the running time of the pulse 
between the object and the antenna, a sequence of pulses is 
used, the receipt of the echo or the return pulse of a 
transmitted pulse being employed to trigger the transmission 
of a second pulse. This process is repeated until a predeter- 
mined number of echo pulses has been received, and the 
time taken to receive a predetermined number of echo pulses 
is proportional to the distance to the object. The method can 
be employed both with electromagnetic signals and acoustic 

GP-PS no. 1,290,915 (Allard & Clark) also discloses a 
distance measurement system based on pulse radar of a 
similar nature to that disclosed in the above-mentioned U.S. 

Otherwise it is well known in the art to measure the 
distance to a movable object by means of a so-called CW 
radar which finds the radial speed of the movable object by 
means of a detected Doppler shift in the return signal or the 
echo signal. By employing a phase comparison it will be 
possible to find the distance to the movable object. 

The disadvantage of using distance measurement systems 
based on detection of a return echo, however, is the many 
sources of error which can arise, e.g. due to false and 
spurious echoes and so-called glitter, i.e. reflected noise 
from ground or sea, and similarly the fact that the natural 
object concerned may not be suited to return an echo, even 
though it could conceivably be equipped with radar reflec- 
tors or a reflecting surface. In various kinds of sport, such as 
ice hockey, however, this is assumed to be an unsuitable 
method. 

Instead of using a passive transponder, i.e. a return echo 
or reflected signals from the natural object, according to the 
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present invention use is preferably made of an active tran- 
sponder. In this case an active transponder should be under- 
stood to refer to a transceiver which is provided in the 
natural object and which, on detecting a polling signal, itself 

5 transmits a response signal. As a rule the transceiver must 
have its own energy source and a pulse transmitter for 
generating the response signal, and in this connection an 
energy source, e.g. in the form of a battery will be a 
substantial drawback, both because the battery can be dam- 
aged and because it will have a limited life. Since the active 

1 transponder also has to be mounted in the natural object, it 
has to be robust and capable of withstanding jolts and shocks 
as well as relative high accelerations. In this context refer- 
ence may be made to the stresses to which, for example, a 
tennis ball, a golf ball or an ice hockey puck will be exposed 

15 when they are struck by, e.g., a racket or club. 

There is therefore a requirement for the active transponder 
to be robust, to withstand considerable acceleration stresses 
and not to require its own energy source, while at the same 
time being small enough to be inserted in a natural object 
which at any rate is not larger than an ice hockey puck. A 
transponder of this kind should also be suitable to be worn 
by, e.g., individuals without causing them any 
inconvenience, such as participants in a game or athletes or 
other people whom it is desirable to record in a television 
picture by means of a synthetic representation of the person 
concerned. This will be discussed in more detail later. 

A block diagram of a transponder as used according to the 
invention in the natural object is illustrated in FIG. 5. The 

30 transponder consists of an antenna 5, an impedance match- 
ing network 6 and a pulse transmitter 7. 

As stated, the transponder's pulse transmitter can be 
implemented as an active unit based on battery operation or 
as a passive system without a battery. In the present 

35 invention, however, an active system without battery is 
preferably employed which has, as already mentioned, a 
number of operative advantages and this is preferably imple- 
mented by means of acoustic surface wave technique (SAW 
technique). 

40 The transponder as illustrated in FIG. 5 and used in the 
present invention is therefore a so-called acoustic surface 
wave component. Such acoustic surface wave transponders 
or SAW transponders consist of a crystal plate with an input 
electrode and one or more output electrodes, as is illustrated 

45 and will be described in more detail in connection with FIG. 
6. 

An SAW transponder T as illustrated schematically in 
FIG. 6, has long been known to those skilled in the art in and 
consists in principle of a crystal, e.g. of lithium niobate with 

so a surface pattern of metal which constitutes transducers, 
reflectors, etc. A polling pulse from the position detector D 
is received by a transducer 8 which is shown in the form of 
a so-called interdigital transducer. The received electromag- 
netic energy in the polling pulse is converted in the trans- 

55 ducer 8 to an acoustic surface wave which moves along the 
crystal. At a certain distance from the transducer 8 there are 
placed a first reflector 9a and a second reflector 9b respec- 
tively. When the acoustic wave strikes the two reflectors 9a 
and 9b, reflection waves are created which move back to the 

6o transducer 8. The transducer 8 will convert the two acoustic 
reflection waves to electromagnetic pulses which constitute 
the response signal which is transmitter via the transpon- 
der's antenna. At the end of the transponder T there may be 
provided a surface wave absorber 10. The path of the signal 

65 is illustrated schematically in FIGS, la and lb. 

When the transponder T receives an interrogator or poll- 
ing pulse 11 of length x p , as illustrated in FIG. la, the pulse 
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transmitter 7, i.e. the transducer 8, is triggered, and a 
response signal is transmitted by the antenna after a specific 
time x 0 . As illustrated in FIG. lb, the response signal 
consists of two pulses 13a, 13b, located at a distance T x apart 
from each other. For different categories of natural objects 
the category of the object may be determined by, e.g., 
measuring the pulse distance T x . in the position detector D 
which also transmits the polling signal. 

In principle this technique is exactly the same as tech- 
niques for the use of radar transponders based on SAW 
technique, since, as is well known, SAW components are 
used as delay members in RF communication and detection 
systems. The transponder, which may be in the form of an 
integrated enclosed chip, can be coded by selecting a puis 
distance between two reflection pulses equal to an ! 
multiple of, e.g., the length x p of the polling puli 
transponder T may, of course, comprise more than the two 
reflectors 8a, 8b as illustrated in FIG. 6, and in this case it 
can be used to emit the response signal with a large number 
of different codes. In this way a transponder T based on 20 
SAW technique can be employed in detection of the distance 
to a large number of natural objects N, since the code thus 
unambiguously identifies which of the natural objects are 
involved. This may be relevant in cases where the method 
and the system according to the present invention are 25 
intended to monitor a game where several objects are 
employed simultaneously in the game, such as in golf or 
during athletics competitions where it is desirable to follow 
a large number of competitors each of whom is equipped 
with a transponder. In the case of approximately simulta- 30 
neous detection of several response signals, in order to 
achieve an unambiguous detection it may be appropriate to 
use special detection techniques, e.g. based on correlation 
between detected, coded reply pulse sequences and pre- 
stored code sequences for each individual transponder. A 35 
second possibility is to use polling signals at different 
frequencies and corresponding frequency-tuned transpon- 
ders. Finally, transponders may be employed with different 
delays x d . In this case the delays must be adapted to the 
distance range concerned for detection, so that the response 40 
signals for the detector in respective unambiguously defined 
times are relative to the time for the transmission of the 
polling signal. 

FIG. la illustrates a pulse pattern employed in the posi- 
tion detector D, where polling pulses 11 are transmitted with 45 
pulse length x p and with a given time interval between the 
polling pulses corresponding to a desired sampling fre- 
quency sequence. The response signal from the transponder 
T will arrive at the position detector D in the form of the 
pulse sequence 13a, 136, as illustrated in FIG. lb, while in 
this case the pulse shape 11 represents an echo caused by 
ground or sea glitter and centred around the real distance to 
a reflection point at or near the transponder T. However, use 
of a SAW component as transponder will delay the response 



added further periods x x for the reply pulses from each 
further reflector 9 in the transponder T. What is now required 
of the system is that the time x should be able to be measured 
with an accuracy which is preferably of the order of ICT 1 m, 
possibly even less, and this will be within the scope of 
current technology. Thus the time delay x d and the period x x 
must be precisely known, and this can be achieved by a 
careful calibration of the SAW transponders used. In order 
to achieve such accuracy, the frequency selected for the 
polling signal should be at the upper end of the L band or in 
the S or X band, i.e. it should have a frequency of between 
circa 1 Mhz and 10 Mhz, or in other words a wavelength of 
between 30 cm and 3 cm. The clock employed in the signal 
processor 1 should have a rate which is adapted to the 
'egral 15 frequency of the polling signal and the response signal. Thus 
— " with present day technology distance detections can be 
obtained with an accuracy of the order of 10" 1 m or less 
within the distances which will normally occur when using 
the method and the system according to the invention, i.e. 
preferably in sports arenas or the like. 

In order to test and calibrate the system according to the 
present invention, in the corner of the area A as illustrated in 
FIG. 1 there can be provided permanently mounted and 
accurately calibrated transponders 1 K in fixed positions and 
coded in order to emit unambiguous identifying response 
signals. These are used to calibrate the system and the 
position detectors D. Transponders are also preferably pro- 
vided both in the position detectors and the cameras. When 
mobile cameras such as "Handycam" and "Steadycam" are 
used, it will be possible to determine the camera positions by 
means of the position detectors D. 

It will also be seen that by using the SAW transponders as 
preferred in the present invention, it will be possible to place 
these without difficulty in most natural objects N which it 
will be appropriate to follow and display in connection with 
a television broadcast, whether it be a football or an ice 
hockey puck. As mentioned, SAW transponders are very 
robust and capable of withstanding substantial acceleration 
stresses, nor do they require their own power source, while 
at the same time they can be designed with extremely small 
dimensions, measuring, for example, only a few millimeters. 
They may, e.g., be enclosed in a shock-absorbing material 
fitted inside the natural object, but otherwise no special 
measures are necessary to protect them from damage. At the 
same time it will be extremely simple to replace them if they 
really should be destroyed. However, they are also so 
inexpensive to manufacture that it will be just as easy to 
replace the natural object with another with a similar SAW 
transponder. When used in connection with other natural 
50 objects than, e.g., footballs, ice hockey pucks etc., for 
example worn on a person, the transponders may be 
designed in the form of an enclosed chip which is attached 
to the person and worn during the period concerned. Since 
the SAW transponder can be arranged to emit an unambigu- 



signal by a predetermined value x d and the reply pulses 13a, 55 ous coded reply signal, the person wearing the transponde 



13b will therefore arrive some time after the glitter pulse 11 
and thereby not be masked by it. The distance selected 
between the reply pulses 13a, 13b here is x x and the delay 
will be equal to x d , the distance from the transducer 7 to the 
reflector in this case naturally corresponding to a running 
time of Vi x d and in addition the distance between each 
reflector corresponding to a running time of V2 x x . The reply 
pulse from the transponder T will now arrive at the detector 
D after an interval x=^c a +x d , wherein x a is the running time 
for an electromagnetic signal from the position detector D to 
the transponder T and back and x d the time delay entered in 
the transponder. To the period x d there will naturally be 



chip will also be unambiguously identifiable. 

The system according to the invention may comprise a 
number of position modules M which are assigned to one or 
more respective television cameras K, each position module 
60 being arranged to cover a predefined area in the x,y,z 
co-ordinate system. This implies that the use of the method 
and the system according to the present invention is not 
restricted to only a well-defined area such as a field for ball 
games or an ice hockey rink and the like, but can be 
65 employed to cover areas of arbitrary shape and size, such as, 
e.g., cross-country skiing courses, Alpine skiing slopes and 
the like. Nor are the method and the system according to the 
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present invention only restricted to sporting arrangements, 
but they can be used to cover events within large and 
relatively freely defined areas. Such areas can be divided 
into several partial areas, and the system does not necessar- 
ily have to cover each partial area or the entire area, but the 5 
use of the system may be restricted to selected partial areas. 

An example of a set-up of the system according to the 
present invention for use, e.g., in a ski run or Alpine skiing 
slope is illustrated in FIG. 8. Here, the area A is divided into 
three partial areas A 1; A 2 , A 3 . To each partial area A lt A 2 , A 3 10 
there are assigned respective position modules M 1; M 2 , M 3 . 
The partial area A 1 is covered, e.g., by a position module M a 
with 3 position detectors D 1; D 2 , D 3 which are intercon- 
nected via the signal line Lj and thereby connected to a data 
processing device Q r The data processing device Qj is 15 
connected via a local data bus B 1 with the cameras K,, K 2 
via their respective regulator modules R 1; R 2 . Typical pic- 
ture fields for the cameras K 1; ICj are indicated by broken 
lines and similarly are surrounded by the main lobe of the 
antenna in one of the position detectors D 3 indicated by 20 
broken lines m D3 . The data processing device Q 5 is con- 
nected to a production unit P via a global data bus B G . The 
same applies to position modules and cameras which are 
assigned to the remaining partial areas Aj, A 3 . Moreover, in 
each of the partial areas A lt A^ A 3 transponders T K are 25 
provided for calibrating in fixed positions, such as at the 
edge of each partial area. Furthermore, the global data bus 
B G may be connected to the timekeeping system, indicated 
in FIG. 8 as CL, at "START" and CL 2 at "FINISH" 
respectively. 30 

In FIG. 8 two natural objects N 1; N 2 are illustrated, the 
first natural object N a being substantially located in the third 
partial area A 3 , and the second natural object N 2 in the 
partial area A r Each of the objects is shown in two different 
positions x,y,z; x',y',z' at the times t and t' respectively, where J5 
it should be understood that positions and times do not need 
to be identical for the two objects N 1; N 2 . The path of each 
of the objects N 1; N 2 between the first and the second 
position is indicated by a broken line. The distances from the 
position detectors D to the objects N-,, N 2 and their positions 40 
are determined as described earlier in connection with FIG. 
1. 

Since the position modules' polling signal has a high 
frequency and directive antennas are employed, it will be 
understood that the signals move substantially in a straight 45 
line. In other words they can be stopped by obstacles in the 
terrain and the like. By dividing up the area A into sub-areas 
A 17 Az, A 3 which are assigned to respective subsystems of 
position modules D and cameras K, the entire area A will 
still be able to be covered by using the method and the 50 
system according to the present invention. The selected 
positions of the position modules D and the cameras K must 
therefore take topographical and other conditions into con- 
sideration. 

In the area A obstacles may be encountered which prevent 55 
the objects N from being shown in the cameras' picture field. 
This may, e.g., be vegetation, groups of people and the like, 
but it will not obstruct the free passage of the detector and 
transponder signals employed. This means that the position 
of the natural object N can still be indicated in the cameras' 60 
picture field, even though the object N at the time concerned 
happens to be covered by obstacles which hinder visual 
contact. In this case the natural object N may naturally be 
represented by a synthetic object S in the correct position 
X,Y in a television picture. Such conditions can easily arise, 65 
e.g., in reports from cross-country skiing, where the skiers 
will be invisible due to the vegetation, even though the 



camera is located in a position which should in theory cover 
the section of the cross-country course concerned. Thus by 
means of the design illustrated in FIG. 8, the methods and 
the system according to the present invention permits a 
complete coverage to be obtained with indication of the 
position of the natural objects N, in the case of sporting 
events or naturally also the position of the competitors, even 
though they are, e.g., masked by vegetation and the like in 
the cameras's picture field. Neither reporters nor viewers 
need, therefore, to be left to guess where an anticipated 
favourite is located at that moment. 

As mentioned above, by means of a method according to 
the invention a synthetic track can be generated in a televi- 
sion picture, the synthetic track thus being intended to 
represent the path of a natural object N in the television 
picture during a given period 8. The path of the natural 
object N can be calculated, as already mentioned, on the 
basis of detected positions x,y,z in a preselected co-ordinate 
system at the time t. The time t will thus lie within a period 
6. The detected positions x,y,z are converted to an X,Y 
position in the picture plane of a relevant television camera 
at the time t. The synthetic track in the television picture is 
created as the connecting line between all X,Y positions for 
the natural object in the picture planes of the television 
pictures which are recorded sequentially during the period 6. 
The synthetic track may now, e.g., be displayed in a still 
picture which need not be recorded during the period 9, or 
it can be continuously updated and generated for each 
individual picture which is recorded during the period 6, 
thus creating the synthetic track cinematographically. Like 
the synthetic object S the synthetic track can be created with 
given, possibly similar attributes with regard to size, shape 
and colour. It may preferably be shown, e.g., as a coloured 
line in a contrasting colour and with indicators for direction 
of movement and possibly also the speed of the natural 
object in the path which corresponds to the synthetic track. 

An example of generation of a synthetic track for a natural 
object N is illustrated in FIGS. 9a, 9b and 9c. FIG. 9a shows 
a path a where the natural object moves from positions p 0 to 
p 3 during the period 9. FIG. 9a illustrates this on the 
projection of the path A in an x,y plane. The path of the 
object N during the period 9 is covered by the camera K and 
naturally recorded as a sequence of individual pictures by 
the cameras K. If camera K has the same setting during the 
period 9, the picture field, e.g., of the camera K in the picture 
plane \ K will show a section A' of the area A, as illustrated 
in FIG. 9b. Here the natural object N is represented by a 
synthetic object S shown in the relevant positions p 0 , . . . and 
with the projection of the natural object's path projected in 
the picture plane \ K as a synthetic track between the various 
positions p 0 , . . . for the synthetic object S as shown in the 
figure. If, e.g., it is a television report from a football match 
which is being shown, N will, naturally, usually be the 
football and the synthetic track will represent the ball's path 
from the position p 0 to the position p 3 , in three dimensions, 
but projected into the picture plane \ K . I^t) may be the final 
picture in the sequence which was recorded during the 
period T, and in this case the track shown for the synthetic 
object S is the path from the beginning of the period 6 up to 
the last recorded picture during 9. It is, of course, not 
essential that the track should only be shown continuously in 
the recorded pictures during the period 6, since the synthetic 
track can be inserted in a freely selected still picture or in a 
television picture which is recorded at a time outside the 
period 9. 

Nor is it a condition that the synthetic track should only 
be shown in natural television pictures, and, as illustrated in 
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FIG. 9c, the actual television picture may be a synthetic 
television picture I A -(S) produced, e.g., by a video generator, 
not necessarily with a view to representing the camera 
perspective, but it may be some kind of graphic represen- 
tation of the area A. FIG. 9c shows a graphic representation 5 
A s of the field A reproduced in an X,Y plane in the form of 
a synthetic television picture. The movement of the natural 
object N is then shown projected in the X,Y plane as the 
movement of the synthetic object S from the position p 0 to 
position p 3 . A person skilled in the art will easily see that it 10 
will be possible to use different perspectives, such as a side 
view, or also possibilities of manipulating the picture with 
the synthetic object and the synthetic track by means of 
various video graphic methods. Nor is there any reason why, 
on the basis of the data acquired for the position and the 15 
movement of the natural object N during the course of the 
game, the path of the natural object should not be presented 
during the course of a game or during a selected period, thus 
aiding the analysis of the game. The technique may also be 
employed by meeting officials and referees in order to judge 20 
situations which otherwise would be difficult to assess if the 
judgement alone were to be based on momentary impres- 
sions during the course of the game. 

As mentioned, there is no reason why the system should 
not detect and monitor a number of natural objects N 1( . . . 2 5 
N,„ the natural objects being unambiguously defined by the 
use of transponders which emit coded reply signals which 
unambiguously identify the natural object concerned. FIG. 
lln refers to a situation which could arise in an area as 
illustrated in FIG. 8. Two natural objects N have moved 30 
through the partial area A 3 , but during different periods. The 
first natural object Nj, e.g. has moved from the position p 0 
to the position p 3 during the period 8, while the natural 
object has moved from the position p' 0 to the position p' 3 
during the period 8'. The paths of the natural objects are 35 
indicated as broken lines in FIG. 9a. In FIG. 9b the detected 
and calculated paths of the natural objects Nj, N 2 are 
represented by the corresponding synthetic objects S 1; S 2 
and converted to synthetic tracks for the objects Sj , S 2 , being 
illustrated simultaneously in one and the same picture I^t) 40 
at the time t. Data for times and speeds which form the basis 
for a comparison between the movement of the objects Nj, 
N 2 , for example Alpine skiers, can be introduced into the 
picture. The X,Y projection of the partial area A 3 in the 
picture plane \ K at the time T will appear as the picture I t (t) 45 
illustrated in FIG. 116 and choice of line and skiing style 
can, e.g. be directly compared. 

It will be obvious to a person skilled in the art that it is 
possible to create a real time reproduction, i.e. the 
development, e.g., of a game or race can be followed in 50 
simulated real time by means of synthetic objects and 
synthetic tracks in the television picture. This may occur, 
e.g. in connection with a playback, but when using the 
method and the system according to the present invention, 
the producer of the television broadcast has a great deal of 55 
freedom to manipulate objects and information as they are 
displayed either in natural or synthetic television pictures. It 
will be possible, e.g., to combine synthetic tracks and 
synthetic objects with various forms of animation graphics, 
if so desired. 60 

As shown in FIG. 1 and FIG. 8, transponders T K are 
installed in various fixed positions in the area A for calibra- 
tion of the system. As mentioned above, transponders may 
also be installed both in the position detectors D and in the 
cameras K. In the latter case movable or mobile cameras can 65 
be employed, i.e. cameras which are not mounted in a fixed 
position, and may be of the type "Handycam" or "Steady- 



cam". These cameras must also naturally be connected to 
data processing devices Q via data buses B, or local data 
buses, in which case they may expediently be based on a 
wireless connection. 

The method and the system according to the present 
invention are not necessarily limited to sports reports, but 
may be applied in a number of other kinds of television 
transmissions. It will also be possible to employ the method 
and the system in transmissions which are not restricted to 
fixed and specific areas, e.g. in connection with nature 
programs on television. If the natural object is located 
beyond the range of the position detectors, but within the 
field of view of a camera, the position of the natural object 
must be determined by other means. In this case the natural 
object may be equipped with a GPS (Global Positioning 
System) receiver for determination of the position, this 
being remotely read at the production location. Similarly, the 
position of a mobile camera, which is also located beyond 
the range of the position detectors, is determined by means 
of GPS and transferred to the production location. Positions 
and paths for the natural object can then be calculated 
according to the method in the present invention, and the 
natural object is represented by a synthetic object which is 
inserted in the correct position in a television picture 
recorded by a camera where the natural object is located in 
the picture field . An example of such an application in nature 
reports may be, e.g., animals which are equipped with radio 
transmitters and GPS receivers and are followed, e.g., by a 
helicopter-borne "Steadycam". Even though the natural 
object may be hidden by obstacles which prevent it from 
being viewed directly in the television picture, its position 
and movement can still be indicated by a synthetic object or 
a synthetic track which is inserted in the television picture. 
In such cases the positioning accuracy is not particularly 
critical and may well amount to several meters, or possibly 
several tens of meters. Such a positioning accuracy lies 
within the range of possibility when using GPS at its highest 
time resolution. 

As stated, in principle the natural objects, which have to 
be detected and displayed represented as synthetic objects 
and possibly represented in motion as synthetic tracks, may 
be practically anything at all. In sports competitions, e.g., the 
competitors may also be equipped with transponders and 
their positions and movement are detected and displayed. 
Since the transponder's reply signal may be in the form of 
an unambiguous code, the identification of the transponder 
will also be unambiguous and the necessary identification 
information can naturally be shown in the television picture 
in some way or other, such as by showing the portrait of the 
competitor overlaid a part of the television picture. 

The use of the method and the system according to the 
present invention in television broadcasts and television 
reports will be capable of being implemented in other 
circumstances than those herein described, and it should also 
be understood that the methods and the system according to 
the present invention for displaying and representing the 
natural object and its movement in the form of a synthetic 
object in a television picture and a synthetic track for the 
synthetic object can be realised in a number of different 
variants and varying attributes which are not expressly 
indicated here, but which nevertheless will be obvious to 
those skilled in the art and which fall within the scope of the 
present invention. 

What is claimed is: 

1. A method for manipulation of at least one movable, 
natural object in a natural television picture, wherein the 
television picture is generated by one or more television 
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cameras, characterized by delecting the distance between the 
object and at least 2 fixed basic positions in a preselected 
x,y,z co-ordinate system at a time t, each basic position 
corresponding to a known position of a detector, determin- 
ing an x,y,z co-ordinate for the object in the preselected x,y,z 5 
co-ordinate system at the time t, determining the distance 
between the camera's lens centre and the object at the time 
t as an object vector in the preselected co-ordinate system, 
determining the television camera's optical axis in the 
preselected co-ordinate system at the time t as a camera 10 
vector in the preselected co-ordinate system, determining a 
line from the television camera's lens centre to the point of 
intersection between the edge of the generated television 
picture and the plane formed between the object vector and 
the camera vector at the time t as a zoom vector, the object 15 
vector being located between the camera vector and the 
zoom vector when the object is visible in the television 
picture at the time t, and when the object is not visible in the 
television picture at the time t, the zoom vector is located 
between the camera vector and the object vector, and deter- 20 
mining an X,Y position of the object referred to the televi- 
sion camera's picture plane and the camera vector on the 
basis of the object vector and the camera vector at the time 
t, and if the object vector is located between the camera 
vector and the zoom vector, to insert a synthetic object in the 25 
X,Y position in the television picture at the time t, the 
synthetic object constituting a representation of the natural 
object recorded by the camera at the time t, or, if the zoom 
vector is located between the camera vector and the object 
vector at the time t, to insert a symbol in the television 30 
picture, the symbol indicating the location of the X,Y 
position of the natural object outside the edge of the picture 
referred to the television camera's picture plane and the 

2. Amethod according to claim 1, characterized in that the 35 
distance between the object and the basic positions, together 
with the object's x,y,z co-ordinate are determined by trilat- 

3. A method according to claim 1, characterized in that 
there are employed 2 fixed basic positions in the preselected 40 
x,y,z co-ordinate system if the natural object is located for 
every value of t in a plane defined by the fact that one of the 
co-ordinates x,y,z of the natural object is equal to zero, or 
that at least 4 fixed basic positions are employed in the 
preselected x,y,z co-ordinate system if the natural object is 45 
located for at least one or some values of t in a space defined 
by the fact that none of the co-ordinates x,y,z is equal to 0, 
thus obtaining an unambiguous determination of the natural 
object's x,y,z co-ordinate at time t. 

4. Amethod according to claim 1, characterized in that the 50 
synthetic object is either created with given attributes for 
size, shape and colour, or that the synthetic object's 
attributes are freely selected, or that the synthetic object's 
attributes are chosen within preselected limits, or that the 
synthetic object's attributes are determined on the basis of 55 
respective reference values for the attributes. 

5. A method according to claim 4, wherein the synthetic 
object's attributes are determined on the basis of respective 
reference values for the attributes, characterized in that the 
synthetic object's attributes are manipulated automatically 60 
on the basis of setting values for the television camera such 

as camera angle and zoom setting and on the basis of the 
colour of the background of the natural object as shown in 
the television picture. 

6. A method according to claim 1, characterized in that the 65 
synthetic object is assigned alphanumerical or symbolic 
signs which indicate the value of one or more parameters of 



the represented natural object, and the parameters may be 
constants which identify the natural object or dynamic 
values such as the object's momentary x,y,z position, dis- 
tance to a freely selected fixed point, course and speed. 

7. A method according to claim 1, characterized in that the 
settings of the television camera such as camera angle and 
zoom setting are controlled in approximate real time on the 
basis of the calculated X,Y position of the natural object at 
the time t and referred to the camera's picture plane and the 
camera vector, with the result that the natural object is 
located within the edge of the picture of the generated 
television picture at any time t, the control being performed 
via a control system assigned to the camera. 

8. A method according to claim 7, wherein more than one 
camera is employed, characterized in that a choice is made 
via the control system as to which camera should be 
employed to generate the television picture which is shown 

9. A method for generating at least one synthetic track in 
a television picture, wherein the synthetic track represents 
the path of a movable, natural object in a natural television 
picture during a given period 9, and wherein a method is 
employed for manipulating the movable, natural object in 
the television picture as indicated in claim 1, characterized 
by calculating the path of the natural object on the basis of 
detected positions x,y,z for the a natural object in the 
preselected x,y,z co-ordinate system at the time t, where te9, 
converting the detected positions at the time t to an X,Y 
position in the television camera's picture plane at the time 
t, and generating the synthetic track in the television picture 
as the connecting line between all X,Y positions of the 
natural object in the picture planes of the natural television 
pictures which are recorded sequentially during the period 8. 

10. A method according to claim 9, characterized in that 
the synthetic track is generated and updated continuously for 
each individual picture which is recorded during the period 
9. 

11. Amethod according to claim 10, characterized in that 
the synthetic track is generated and displayed as the path of 
the natural object during the period 6 in a freely selected 
television picture which forms a background for the natural 
object, the X,Y positions being scaled for each time te6 and 
assigned to the picture plane of the freely selected television 

12. A method according to claim 11, characterized in that 
the freely selected television picture is a natural television 
picture, the freely selected television picture forming part of 
the sequence of the natural television pictures which are 
recorded during the period 0, or that the freely selected 
television picture is a synthetic television picture. 

13. A method according to claim 9, characterized in that 
the X,Y positions for one or more of the times t are indicated 
on the synthetic track, the indication of the X,Y position in 
the synthetic track being formed as a synthetic object with 
given attributes for size, shape and colour, and in such a 
manner that the synthetic object constitutes a representation 
of the natural object recorded by the camera at the time te8. 

14. A method according to claim 13, characterized in that 
the indication of the X,Y positions or the synthetic object is 
assigned alphanumerical or symbolic signs which indicate 
the value or one or more parameters for the represented 
natural object, and the parameters may be constants which 
identify the natural object or dynamic values such as the 
object's momentary x,y,z position, distance from a freely 
selected fixed point, course and speed at the time te0. 

15. A system for implementing the method for manipu- 
lation of at least one movable, natural object (N) in a natural 
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lelevision picture, wherein the television picture is generated 
by one or more television cameras (K), together with imple- 
mentation of the method for generating a synthetic track in 
a television picture, wherein the synthetic track represents 
the path of a movable, natural object in the television picture 
during a given period 9, and wherein a method is employed 
for manipulating the movable, natural object in the televi- 
sion picture, characterized in that the system comprises a 
transponder (T) provided in the natural object (N) and 
arranged to react to an optical, acoustic or electromagnetic 
signal received by the transponder with transmission of a 
response signal, at least one position module (M) with at 
least 2 position detectors (D) for transmitting optical, acous- 
tic or electromagnetic signals and receiving response signals 
from the transponder (T) and provided in respective basic 
positions in a preselected x,y,z co-ordinate system, together 
with a signal processor (1) arranged to determine the dis- 
tance between a position detector (D) and the object (N) at 
a time t, a computing module (2) connected with the signal 
process (1) and arranged to calculate the x,y,z co-ordinates 
for the object (N) in the preselected co-ordinate system at the 
time t and on the basis of the calculated x,y,z co-ordinates 
for a number of times t to calculate a path for the object (N), 
together with an object vector (V w ) given by the distance 
between the camera's (K) lens centre and the object (N) at 
the time t, a camera vector (V K ) given by the camera's (K) 
optical axis at the time t and a zoom vector (VJ between the 
camera's lens centre and the point of intersection (Z) 
between the edge of the picture and the plane formed 
between the object vector (V^) and the camera vector (V K ) 
at the time t, the object vector (V w ) being either located 
between the camera vector (V^.) and the zoom vector (V z ) 
or the zoom vector (V z ) between the camera vector (V K ) and 
the object vector (V^,), a camera control system (3) con- 
nected with the computing module (2) and arranged to detect 
or generate values for the camera settings, and a manipulator 
module (4) connected with the camera control system (3) 
and the computing module (2) arranged to a) create a 
synthetic object (S) in an X,Y position for the natural object 
(N) in the recorded television picture at the time t, the 
synthetic object )S) constituting a representation of the 
natural object (N) recorded by the camera at the time t or b) 
create a symbol n the television picture, the symbol indi- 
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eating the X,Y position for the natural object outside the 
edge of the picture at the time t, c) generate and select 
attributes for the synthetic object (S), or d) generate at the 
time t a synthetic track in a recorded or generated television 
5 picture, the synthetic track representing the path for the 
natural object (N) during a period 9 before or up to the time 

16. A system according to claim 15, characterized in that 
the system comprises a number of position modules (M) 

10 each of which is assigned to one or more respective televi- 
sion cameras (K), each position module (M) being arranged 
to cover a predefined area in the x,y,z co-ordinate system. 

17. A system according to claim 15, characterized in that 
the position detector (D) comprises a microwave transceiver 

15 and an antenna (II) for transmission and receipt of micro- 
wave signals, and that there is further provided a transponder 
T in each of the position detectors (D) and/or in each of the 
cameras (K) respectively. 

18. A system according to claim 17, wherein each said 
20 position module comprises at least 4 position detectors 

(Dl, . . . D4) for unambiguous determination of the x,y,z 
co-ordinates for the object (N) at the time t. 

19. A system according to claim 15, characterized in that 
the transponder (T) is either a passive transponder, or an 

25 active transponder, the active transponder being a surface 
wave component (SAW component), and that the response 
signal from the active transponder (T) in each case consti- 
tutes a code which unambiguously identifies the transponder 
by delaying the response signal from the active transponder 

30 (T) by a predetermined value x d in relation to the time of 
receipt of a signal which causes the response signal to be 
transmitted form the transponder. 

20. A system according to claim 15, characterized in that 
the camera control system (3) comprises a control loop for 

35 automatic control of camera settings in approximate real 
time, the camera settings being influenced via a regulator 
module (R) provided on the camera (K) which module 
further comprises sensors for detection of the camera 
settings, and/or that the manipulator module (4) comprises a 

40 device for automatic generation of attributes for the syn- 
thetic object. 
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CROSS-REFERENCE TO RELATED 
APPLICATIONS 

This application relates to U.S. applications Ser. No. 
07/692,718 filed on Apr. 29, 1991 entitled "TRAFFIC 
FLOW MEASURING METHOD AND APPARATUS" 
issued as U.S. Pat. No. 5,283,573 on Feb. 1, 1994, Ser. No. 
08/018,558, entitled "Traffic Flow Measuring Method and 
Apparatus" filed Feb. 17, 1993, as a continuation of the 
above-identified application, and Ser. No. 07/913,929 filed 
on Jul. 17, 1992 entitled "IMAGE RECOGNITION 
METHOD AND IMAGE RECOGNITION SYSTEM", 
assigned to the present assignee. The contents of these 
applications are incorporated herein by reference. 



BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to an object recognition 
system suitable for grasping motions of an object or more in 25 
particular to an object recognition system suitable for track- 
ing a moving vehicle or the like, on the one hand, and to a 
system for detecting an abnormal phenomenon on a road or 
the like, or more in particular to a system for processing and 
detecting an image on a TV camera. 30 

2. Description of the Related Art 

To recognize the movement or the like motion of an object 
by use of image processing is considered to provide a very 
effective means in various applications. For example, 3 _ 
vehicles running on the road are recognized to measure the 
information such as the number and speed of vehicles 
passed, whether they are stationary or not, etc. 

Conventional systems for recognizing vehicles by pro- 
cessing an image on the TV camera are described in "Vehicle 40 
Recognition by DTT Method", Computer Vision, Informa- 
tion Processing Society of Japan, 72-5, May 1992. 

To detect the traffic condition is very effective in main- 
taining a smooth road traffic. Systems including what is 
called a loop-type vehicle sensor and an ultrasonic vehicle 45 
sensor have been used for detecting the traffic condition. 
These systems exert ultrasonic wave or magnetism at a 
ground point on the road, measure the existence of a vehicle 
according to the change thereof, and detect the number and 
speed of vehicles on the basis of the time of change. These 50 
systems, however, are basically capable of determining the 
traffic condition only at a single ground point and therefore 
are disadvantageous in measuring a wide range of condi- 
tions. For this reason, a method has positively been used 
recently, in which an image obtained from the TV camera is 55 
processed to measure the traffic condition, as described in 
JP-A-2-122400. According to the conventional system dis- 
closed in JP-A-3-204783, on the other hand, a moving object 
is traced by center-of-gravity calculation of a binary-coded 
input shade image from the TV camera. Another conven- 60 
tional system disclosed in JP-A-62-1 80488 concerns char- 
acter recognition but not the recognition of a mobile object. 
According to the last-mentioned method, a multi-valued 
template is prepared and decomposed into a plurality of 
binary templates, so that similarity between the binary 65 
template and a binary-coded input image is determined by 
pattern matching thereby to achieve character recognition. 
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The prior art relating to pattern matching is disclosed in 
JP-A-63-98070, etc. 

Further, early detection of an abnormal phenomenon on 
the road is important in maintaining a smooth road traffic. 

Specifically, it is necessary to detect an accident, a sta- 
tionary vehicle, a fallen object or the like at an early time and 
prevent the secondary damage from being caused by such an 
abnormal phenomenon. Detection of an abnormal phenom- 
enon in a tunnel is especially important. Systems applicable 
to such a purpose are expected to be developed more and 

According to the conventional image processing systems, 
however, only what is called "the traffic flow data" including 
the number and speed of vehicles is measured, but the 
configuration thereof lacks means to detect various abnor- 
mal phenomena. An example of such a conventional traffic 
flow measuring system is disclosed in "Architecture of 
Traffic Flow Measuring System Using Image Processing" in 
a paper for Lecture at the 37th National Convention of 
Information Processing Society of Japan, 6T-6, 1988. 

In the "Vehicle Recognition Using DTT Method" 
described above, an input image is differentiated and binary- 
coded, a binary projection distribution along X axis (hori- 
zontal direction) of this binary image is determined, and 
only the coordinates of this projection distribution beyond a 
predetermined threshold value are stored, thus determining 
the trace of vehicles. This process has been conventionally 
employed in most cases of measuring the number and speed 
of vehicles by image processing, thereby posing the problem 
that it is difficult to set a binary-coded threshold value on the 
one hand and measurement is difficult when vehicles are 
superposed one on another on the other. 

According to the conventional techniques disclosed in 
JP-A-2-122400, JP-A-3-204783, JP-A-62-180488 and 
JP-A-63-98070, the number and speed of vehicles are mea- 
sured by image processing in most cases through the pro- 
cesses of differentiation of input image, binary-coding and 
feature measurement. The problem of these methods is that 
a binary-coded threshold value cannot be easily set and 
measurement is difficult for vehicles superposed. Also, the 
conventional technique for binary-coding and center-of- 
gravity calculation of an input shade image encounters the 
problem that the image contrast is reduced by the change in 
the environment or situation in which the system is installed, 
thereby making it sometimes impossible to discriminate a 
vehicle from the background. The decomposition of a multi- 
valued template into a plurality of binary templates for 
pattern matching fails to recognize a moving object accu- 
rately. 

As for abnormal phenomena in a tunnel, TV cameras are 
not actually installed at sufficiently short intervals to monitor 
the entire area in the tunnel. No one can predict where an 
abnormal phenomenon occurs. According to the conven- 
tional traffic flow measuring functions, therefore, it is vir- 
tually impossible for the conventional traffic flow measuring 
functions alone to measure abnormal phenomena occurring 
outside of the visual field of TV cameras. Another disad- 
vantage of the conventional systems is that all abnormal 
phenomena cannot be grasped with the data on traffic flow. 

SUMMARY OF THE INVENTION 

According to a first aspect of the present invention, there 
is provided an object recognition system comprising a shade 
template memory circuit storing images at specified points 
of an object, a shade pattern matching circuit matching 



Evidence Appendix 
Appellant's Brief, page 132 



patterns between the shade template and an input image, and 
an integral processing section determining the position of an 
object by searching for an input image analogous to the one 
of the shade template, defining the speed and direction of 
vehicle movement by the coordinate change of the position 5 
of the object, and identifying that templates identical in 
behaviour represent the same object, wherein a unique area 
or feature area is picked up from an image of an object, the 
unique image is registered as a shade template, a point of 
movement is determined for each template by shade pattern 10 
matching, the speed and direction of vehicle movement are 
determined from information on the determined points of 
movement, and the result of the determination is integrated 
to recognize the whole of moving objects. 

This technique will hereinafter be referred to as "Partial 15 
Correlation Integration Method (PCIM)". 

According to this method, even for an object of low 
contrast such as in the case of a black vehicle located in a 
shadow environment, for example, portions of the object 
high in contrast can be registered as a shade template and 
integrated by post-processing. As a result, a moving object 
can be followed or tracked in satisfactory manner in spite of 
a change in brightness or superposition of objects which 



According to a second aspect of the present 
there is provided an object recognition system further com- 
prising a template updating section whereby shade templates 
used by being extracted from an input shaded image are 
sequentially updated with the movement of an object to be 3Q 
detected, so that the shade templates are sequentially 
updated and therefore moving objects can be followed even 
when the shape or the like thereof undergoes a change over 
a long period of time. 

According to a third aspect of the present invention, there 35 
is provided an abnormality detection system, in which 
"traffic flow measurement", "detection of stationary 
vehicles", "detection of abnormally-running vehicles" and 
"measurement of congestion degree" are executed within the 
visual field of TV cameras, and the result of these measure- 40 
ments is interpolated in space and time thereby to detect an 
abnormal phenomenon in other than the visual field. 

Further, in order to determine various abnormal phenom- 
ena accurately, the various functions mentioned above are 
judged integrally or synthetically. 45 

By way of explanation, the "traffic flow measurement" is 
for measuring the speed and number of vehicles in the visual 
field of TV cameras, the "detection of abnormally-running 
vehicles" for monitoring the running pattern of vehicles in 
the visual field of TV cameras, the "detection of stationary 50 
vehicles" for detecting a vehicle stationary or out of order or 
a fallen object within the visual field of TV cameras, and the 
"measurement of congestion degree" for stepwise measure- 
ment of the degree of vehicle congestion. An abnormality 
judgement is made by considering these factors as a whole. 55 
The above-mentioned functions are measured at each 
ground point, and the resulting data are spatially interpolated 
to predict an abnormality outside of the visual field of TV 



Accurate detection of various abnormal phen 
thus made possible, and any abnormal phenomenon outside 
of the visual field of TV cameras can also be detected. 

BRIEF DESCRIPTION OF THE DRAWINGS 

6 

FIG. 1 is a block diagram for explaining an embodiment 
of the present ir 



FIGS. 2A and 2B are illustrations showing an example of 
processing of shade pattern matching. 

FIGS. 3A and 3B are illustrations showing an example of 
processing of partial shade pattern matching. 

FIG. 4 is a diagram showing traces of "time" and "ordi- 
nates" obtained by partial shade pattern matching. 

FIG. 5 is a diagram-showing traces of "time" and 
"abscissa" obtained by partial shade pattern matching. 

FIG. 6 is a diagram showing a model of an object image 
with moving objects followed by the partial correlation 
integration method. 

FIG. 7 is a diagram showing traces of "time" and "ordi- 
nates" obtained by processing according to the partial cor- 
relation integration method. 

FIG. 8 is a diagram showing a search area limitation 
method applied in following a moving object. 

FIG. 9 is a diagram for explaining the relation between the 
speed of a moving object and the range of direction in which 

FIGS. 10A and 10B are diagrams for explaining the 
search area for each template applied in following a moving 

FIG. 11 is a diagram for explaining a template size 
determination method with a bird's eye view of a road. 
FIG. 12 is a diagram for explaining the vehicle movement 



FIG. 13 is a diagram showing an example of a shade 
pattern matching circuit used for updating the template. 

FIGS. 14A and 14B are illustrations showing the process- 
ing executed when vehicles are running in parallel. 

FIG. 15 is a diagram showing traces of "time" and 
"ordinates" plotted when vehicles are running in parallel. 

FIG. 16 is a diagram for explaining a method of assuring 
satisfactory processing even when vehicles are running in 
parallel. 

FIG. 17 is a diagram for explaining traces of "time" and 
"ordinate" plotted using a method of assuring satisfactory 
processing even when vehicles are running in parallel. 

FIG. 18 is a diagram for explaining traces of "time" and 
"abscissa" plotted using a method of assuring satisfactory 
processing even when vehicles are running in parallel. 

FIGS. 19A and 19B are illustrations for explaining a 
technique for following a vehicle by correlation calculation 
on the basis of a shade projection distribution. 

FIGS. 20A and 20B are illustrations showing a method of 
following a vehicle along abscissa with correlation calcula- 
tion on the basis of a shade projection distribution. 

FIG. 21 is a diagram for explaining another embodiment 
of the present invention. 

FIGS. 22A and 22B are illustrations showing an example 
of the processing of shade pattern matching. 

FIGS. 23A and 23B are diagrams showing the coordinates 
detected by shade pattern matching. 

FIGS. 24A to 24D are diagrams for explaining the manner 
in which a moving object is followed. 

FIG. 25 is a diagram showing an example of main flow in 
following a moving object. 

FIG. 26 is a diagram showing an example of the flow of 
vehicle search processing in a detection area with a moving 
vehicle followed. 

FIG. 27 is a diagram showing an example of the flow of 
following a vehicle as a moving object. 
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FIG. 28 is a diagram showing an example of processing 
for minimizing the number of templates. 

FIG. 29 is a diagram showing another example of pro- 
cessing for minimizing the number of templates. 

FIG. 30 is a diagram showing the processing for mini- 5 
mizing the number of templates. 

FIG. 31 is a diagram showing an example of preparation 
of an initial template image of a vehicle. 

FIG. 32 is a diagram showing the condition of a vehicle i 0 
displaced from an extracted area. 

FIG. 33 is a diagram for explaining a vehicle set at the 
center of an extracted area. 

FIG. 34 is a diagram for explaining the manner in which 
the size of a template image is determined in accordance 15 
with the vehicle size. 

FIGS. 35A and 35B are illustrations showing a method of 
updating a template with the vehicle size changing. 

FIG. 36 is a block diagram showing the configuration of 2Q 
an abnormality detection system according to an embodi- 
ment of the present invention. 

FIG. 37 is a flowchart for explaining the outline of 
processing of traffic flow measurement. 



FIG. 1 diagrammatically shows a configuration of a 
system according to the present invention. An image from a 
TV camera 1 or the like is applied to an image memory 3 
through an A/D converter 2 for converting an analog data 
into a digital data. The image memory 3 is a shade image 
memory of about 8 bits. A shade template memory circuit 4, 
on the other hand, is for storing shade templates of a shaded 
image. The shade image memory 3 and the shade template 
memory circuit 4 are configured to scan the interior of the 
memory by an address processor 5. The shade template 
memory circuit 4 is for storing a template area cut out of an 
image through a template extracting circuit 8. The shade 
template memory circuit 4 also has a plurality of shade 
templates for storing various shade images. A shade pattern 
matching circuit 6 is for determining the matching between 
the image data 30 of the image memory 3 and the image data 
40 of the shade template memory circuit 4, and for executing 
the normalized correlation calculation of a normal level or 

In normalized correlation of an input image f(x, y) against 
a shade template T(p, q), the similarity r(u, v) of the point of 
input image f(u, v) is given as 



'}]" 



hl>*<»<>'-{ 14*""" ! 1 ii™ ) 1 



FIG. 38 is a diagram for explaining the outline of pro- 35 
cessing of monitoring abnormally-running vehicles. 

FIGS. 39A, 39B and 39C are diagrams for explaining an 
example of data accumulated as obtained for monitoring 
abnormally-running vehicles. 

FIG. 40 is a flowchart for explaining the outline of 40 
processing of detecting a stationary vehicle. 

FIG. 41 is a diagram for explaining the outline of pro- 
cessing of congestion degree measurement. 

FIG. 42 is a diagram for explaining an example of the 45 
principle of spatial interpolation between cameras. 

FIG. 43 is a diagram for explaining an example of the 
principle of time interpolation between cameras. 

FIG. 44 is a block diagram showing an example of 
hardware configuration of an abnormal detection system 50 
according to the present invention. 

FIG. 45 is a block diagram showing a specific configu- 
ration of an image processing unit in FIG. 44. 

FIG. 46 is a block diagram showing a specific conftgu- 
ration of a central overall decision section in FIG. 44. 3 



An embodiment of the present invention will be described 
with reference to the accompanying drawings. By way of 
explanation, a road is photographed by TV camera with the 
description made about a technique of recognizing vehicles 
running on the road. This technique is applicable not only to c 
measurement of behaviour of not only vehicles but all other 
objects. 



where p, q are x, y sizes of a gray scale or shade template, 
respectively. Any method of determining a correlation may 
be used to the extent that an effect similar to the above- 
mentioned method is obtained. 

The operation of the image memory 3, the scanning of the 
shade template memory circuit 4 and the operation of the 
shade pattern matching circuit 6 are all controlled by a CPU 
(not shown). More specifically, the address processor 5 and 
the shade pattern matching circuit 6 are activated by the 
CPU, the image address given in Equation (1) is generated 
at the address processor 5, a data is read from the image 
memory 3 and the shade template memory circuit 4, a data 
is determined as required for similarity calculation at the 
shade pattern matching circuit 6, and the similarity r(u, v) is 
calculated at the CPU. A coordinate with a large similarity, 
once determined, is stored in a trace management table 9 for 
managing the "time", "abscissa" and "time", "ordinate". 
Then, the isolation/integration processing is effected by an 
isolation/integration processing circuit 10 against the trace 
management table. To explain the shade pattern matching 
circuit 6 briefly, the data relating to a shade template is 
capable of being calculated in advance, and therefore the 
data relating to an input image is calculated and transmitted 
to the CPU at the time of calculating a similarity. The data 
relating to an input image is determined by the calculation 
according to Equation (4) from Equation (2) through the 
shade pattern matching circuit 6 in the above-mentioned 
case. The same operation may be shared by the shade 
template memory circuit 4 and the image memory 3. 
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-continued 
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An example of shade pattern matching of an input image 
is shown in FIGS. 2A and 2B. In the case where a vehicle 
20 is displayed on an input image f(t) at a time point t, a 
shade template Tl is cut out of the input image f(t), raster 10 
scanning is effected at the same time as pattern matching of 
an input image f(t+dt) at time point t+dt, and a coordinate is 
sought with an increased similarity r(u, v). Then, the maxi- 
mum point of similarity is determined as shown in FIG. 2B. 
As a result, the instantaneous speed and direction of vehicle 15 
movement can be determined from the maximum similarity 
point of time point t+dt and the vehicle existence coordinate 
(central coordinate of template) at a time point t. 

Incidentally, the horizontal axis of an image memory is 
referred to as X axis, and the vertical axis thereof as Y axis. 20 

The advantage of the above-mentioned normalized cor- 
relation processing is that to the extent that a registered 
shade template and an image pattern are similar to each 
other, a similarity to some degree is obtainable even when 
the brightness undergoes a change or an object is hidden to 25 
some degree by something (as when vehicles are superposed 
on two-dimensional display or when only a portion of a 
vehicle is visible due to a shadow of a building or the like). 
More specifically, even though similar to the road surface 
brightness, an object can be recognized as a vehicle as long 30 
as it has a vehicular shape. According to the conventional 
systems using binary-coding at a certain threshold value, 
recognition of a vehicle of low contrast is very difficult. The 
technique according to the present embodiment, on the other 
hand, facilitates extraction of a vehicle. 35 

Vehicles to be recognized are of various sizes, types and 
colors. The processing by pattern matching is accompanied 
by the problem of size change. Pattern matching techniques 
disclosed in U.S. application Ser. No. 07/789,682 filed Nov. 
8, 1991 are applicable to the present invention. The contents 40 
of that application is incorporated by reference herein. As far 
as the pattern matching of whole vehicles is executed, the 
recognition of vehicle size is required in advance, thereby 
making it difficult to apply the system to vehicles passing at 
random. When a human being observes the movement of an 45 
object of considerable size, therefore, the whole of the 
particular object is not always followed, but the movement 
of a specified point thereof is followed, so that points where 
the movements are identical in speed and direction are 
finally integrated to recognize a large object. In other words, 50 
a plurality of partial templates smaller than a vehicle are 
prepared, and a shade pattern matching is effected for each 
template. The templates, for which shade patterns are 
matched and the resulting speeds and directions, are judged 
to be similar to each other and are integrated to represent the 55 
same vehicle independently of the size. In the case where no 
moving object exists in a registered template (in the case of 
an image of the road surface alone), the maximum point of 
similarity cannot be defined or displaced from the central 
coordinate of a registered template. 60 

As will be described in detail with reference to FIG. 3, 
partial areas smaller than the vehicle size are set in an input 
image. In FIGS. 3A and 3B, templates Tl and T2 are set. 
These partial areas are registered as templates respectively, 
and matched with sequentially applied images to determine 65 
the maximum point of matching for each template. If this 
condition is expressed as "time" along abscissa and as 
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"ordinate" along vertical axis as shown in FIG. 4, a trace can 
be determined for each template, where the unit along time 
axis corresponds to a sampling interval of image input. 
When an image is introduced for every three frames, for 
example, the unit is 3x1/30=100 ms. This management is 
effected at a trace management table 9. In view of the 
delicate difference in matching point with each template, the 
moving speed (displacement of ordinate) undergoes a 
change. Nevertheless, substantially similar behaviours are 
obtained due to the fact that the partial templates are for the 
same vehicle. As a result, an integration processing of these 
templates makes it possible to execute the recognition of a 
whole vehicle. An integration processing is executed at an 
iteration processing circuit 10. FIG. 5 is a diagram showing 
traces plotted as "time" along abscissa and "X coordinate" 
along ordinate. The discrimination of straight run and lane 
change is made possible by recognizing the format of the 
traces. FIG. 6 shows a detailed case involving two passing 
vehicles, in which images inputted at intervals of a certain 
unit time over a period from tl to tl6 are processed. 
Templates carry shade images of three partial areas at the 
lower part of the screen entered by the vehicle. 

The processing steps will be briefly explained below. 

(1) Three partial templates Tl(tl) to T3(tl) for the front 
part of a vehicle are registered in the image of time point tl, 
where t designates time, and T a template. 

(2) The shade pattern matching is effected for the image 
of time point t2 by use of the registered templates Tl(tl) to 
T3(tl). As a consequence, moving traces are determined for 
the partial templates Tl(tl) to T3(tl). 

(3) Three partial templates Tl(t2) to T3(t2) are registered 
also for the image of time point t2. 

(4) In similar fashion, the shade pattern matching is 
effected for the image of time point t3 using the templates 
Tl(tl) to T3(tl) and Tl(t2) to T3(t2). As a result, moving 
traces are obtained for the partial templates Tl(tl) to T3(tl) 
and Tl(t2) to T3(t2). 

(5) The above-mentioned steps of processing are sequen- 
tially repeated. In other words, the registration of partial 
templates at a given time point and the shade pattern 
matching between the partial templates registered at the 
preceding time points and the present image are sequentially 
repeated in parallel. The traces for respective partial tem- 
plates are thus obtained as shown in FIG. 7. Traces for the 
same vehicle are located close to each other in distinction 
from other vehicles. The interval between two groups of 
close traces represents the distance between two vehicles 
running in the same direction, and the width of each trace 
group the vehicle length. The inclination of the traces, on the 
other hand, represents the vehicle speed. 

(6) The partial templates are separated and integrated. In 
the case of FIG. 7, for example, the partial templates Tl(tl) 
to T3(tl) are processed on the assumption that they represent 
the same object. 

It will be seen that the vehicle behaviour recognition is 
easily executed by using this technique. 

A specific example of each process will be explained 

1. Method of separation and integration processings 

The separation and integration processings after deter- 
mining the moving point of each template may be executed 
by the technique as described below. 
1.1 Integration processing 

(1) The trace of each template is approximated by curve 
according to the least square method or the like, and those 
partial templates having small mutual distances are inte- 
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(2) The knowledge about an object image (relating to the 
concentration distribution, shape, etc.) is prepared, and 
determination is made as to which part of the object is 
represented by the shape of a given template. In this way, all 
the templates are integrated. In the case of a vehicle, for 5 
example, assume that a template 1 represents "the left image 
of the tail lamp", a template 2 "the right image of the tail 
lamp", and a template 3 "the central image of the rear 
window". This knowledge permits the determination that the 
template 2 is positioned to the right of the template 1, and 1( 
the template 3 is arranged above them. In other words, the 
templates 1 to 3 are determined to represent the same object. 

The size of a template is such that initial partial templates 
may be processed always as such (in the form of a hypo- 
thetical large template) and, after integration, as a template 
covering a rectangular area containing all the integrated l; 
partial templates. 
1.2 Separation processing 

(1) In the case where the feature amount (for example, 
average concentration) of some of the integrated templates 
suddenly increases, they are separated from the integrated 21 
pattern. 

(2) The templates thus integrated are assumed to be 
coupled under a certain resistance. In the case where the 
behaviour of a given template is different from that of 
another template connected thereto, therefore, they are left 2: 
integrated if the behaviour is not to such a degree as to cut 
off the resistance. If the difference is so great as to cut off the 
resistance, on the other hand, the affected template is sepa- 
rated. The resistance may be denned as a weight represent- 
ing the strength of coupling or a function representing the 3 
degree of coupling. 

2, Method of limiting search range 

In recognizing the behaviour of a vehicle running along 
the road, full scanning of the correlation calculation of 
registered templates is not always necessary. More specin- 3 
cally, in view of the fact that the range of movement of a 
vehicle is limited to some extent, the next range of search 
can be specified by the past behaviour of the vehicle. The 
processing time of the similarity calculation is proportional 
to the number of picture elements (the sizes of templates and 4 
search range), and therefore, the search range should be 
reduced as far as possible. An example is shown in FIG. 8. 
In the case where a vehicle moves from the coordinate (xO, 
yO) to (xl, yl), for instance, it is possible to specify the next 
search area by using the moving speed V and the direction 1 
of movement 9. A predicted position is included in a 
fan-shaped range shown in FIG. 8 as determined from the 
minimum speed Vmin, the maximum speed Vmax and the 
amount of change in the direction of movement 2<j> against 
the preceding moving speed V. To simplify the processing, : 
the rectangular area surrounding the fan-shaped area is used 
as the next search range. Such a search range is determined 
according to Equations (5) to (8) below. 



I5=il+Vmil 



os(6+<|>) 



ys=yl+Vmax-sin (S-HjO 



>'e=yl+Vniinsin (6-(j>) (8) 60 

where (xs, ys) represents the starting coordinate and (xe, ye) 
the final coordinate of an area. The range <j> of the vehicle 
direction may be changed as a function of the speed V as 
shown in FIG. 9. Assume that the object of recognition is 65 
other than a vehicle which is simple in behaviour pattern, or, 
that a human being, an animal or the like object is to be 
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recognized and followed, for example, a simple search range 
as determined above is sometimes insufficient. In view of 
this, the past behaviour pattern is learned by a neural 
network in order to specify the next mobile range from the 
traces of past movement. If this behaviour pattern is used for 
prediction of the future behaviour, the resulting search range 
which will be more detailed is expected to reduce the 



determined at a time point t as shown in FIG. 10A, a search 
area can be determined as shown in FIG. 10B, thus remark- 
ably reducing the processing time. 
3. Method of updating templates 

In spite of the fact that the foregoing description concerns 
the case in which a road is viewed from right above, it is 
almost impossible actually to photograph a road from over- 
head. A bird' s-eye view of a road is the result. In such a case, 
a vehicle appears smaller with the ascendance in the screen 
as shown in FIG. 11. In the method of following a vehicle 
according to the above-mentioned correlation calculation 
method, therefore, a vehicle located at a far point cannot be 
detected and therefore cannot be followed, although detec- 
tion is possible of vehicles appearing to be of the same size 
as the registered shade templates. This problem never fails 
to arise in applications involving a wide monitor range such 
as in monitoring human intruders or watching railroad 
crossings. 

In view of this, a shade template is updated while fol- 
lowing the movement of vehicles as shown in FIG. 11. More 
specifically, in the case where templates with the ordinate 
centered at Yl is registered from the image of time point tl, 
assume that a similarity point is detected at the ordinate Y2 
by shade pattern matching of the image of time point t2. The 
template size in registration is reduced in accordance with 
the ratio determined by the coordinate Y2, and the shade 
image at time point t2 in the vicinity of the similarity point 
is registered again as a template. The shade template newly 
registered is used for matching in the next search area. 
Sequential executions of this process permits the versatile 
operation of following vehicles against changes in vehicle 
size. This method has been developed on the basis of the fact 
that the size and direction of movement of a vehicle does not 
substantially change instantaneously but slowly. 

A template may be prepared, instead of by the above- 
mentioned method using a latest input image, alternatively 
according to the average ((f+g)/2) of the preceding template 
image (g) and an image newly detected in the vicinity of a 
vehicle position (f) or a linear coupling calculation (af+Pg, 
a+P=l). 

Assume that the number of vehicles turning to the right at 
an intersection is measured by using the above-mentioned 
technique as shown in FIG. 12. In view of the fact that the 
vehicles turning to the right change the direction thereof 
gradually, the following of a vehicle ceases midway with a 
single shade template. The following of vehicles, however, 
is facilitated by updating the shade template at the same 
time. Also, the movement of a human being constantly 
changing in shape is capable of being followed by updating 
a template as described above. 

When a shade template is updated, however, the data 
relating to the shade template in the calculation of Equation 
(1) mentioned above cannot be determined in advance. This 
value is thus required to be determined with rapidity. In view 
of this, a circuit may be added as shown in FIG. 13 for 
acquiring the data on the shade template T (i, j), so that with 
the registration of a shade template, the calculations of 
Equations (9) and (10) are executed to always permit 
high-speed processing. 
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(10) 



where p, q are constantly variable with the size of the shade 
template. 

A template is extracted from an input image by a template 
extracting circuit 11. The next template is prepared at a 
template update circuit 12 by averaging or linear coupling of 10 
the template thus extracted and the preceding template. The 
resulting template is registered and used for the next opera- 
tion of following vehicles. A device for continuous follow- 
ing operation is thus configured. 

4. Size of template 15 
Any case of vehicles running in parallel is not described 

above. Actually, however, a road often has two or three 
lanes, and therefore, it is necessary also to be able to process 
the recognition of vehicles running in parallel. A case of 
recognition of the behaviour of vehicles running in parallel 20 
is shown in FIGS. 14A and 14B. Templates Tl to T4 are 
registered with respect to an input image as shown in FIG. 
14A. When these templates are used to follow the vehicle 
behaviour by shade pattern matching against an input image 
of time point t+dt, a matching point is detected for each 25 
template. If a moving trace is plotted as "time" along 
abscissa and "ordinate" along vertical axis, the diagram of 
FIG. 15 is obtained. It will be noted that in the case where 
the speed and direction of movement of two vehicles are 
substantially identical to each other, mere analysis of the 30 
traces shown in FIG. 15 cannot discriminate small and large 
vehicles. 

This is due to the fact that the area on a road separating 
the two vehicles fails to be recognized. More specifically, a 
template exclusively covering the road cannot be registered, 35 
if each template is excessively large in size. It is, therefore, 
important to set the template width to not more than the 
vehicle intervals (along X direction). The operation with a 
reduced template size is shown in FIG. 16. One or more 
templates (corresponding to T5 in the case under consider- 40 
ation) of a size smaller than the interval between vehicles are 
always provided and are followed with the shade pattern 
matching described above. The time-ordinate and time- 
abscissa relationships in this state are shown in FIG. 18. In 
the time-ordinate graph, which templates are associated with 45 
the same vehicle is unknown when two vehicles are running 
at substantially the same speed. From the observation of the 
time-abscissa graph, however, an area lacking a vehicle is 
detected, and therefore the templates Tl to T4 and T6 to T10 
are known to belong to the same vehicle. 50 

As explained above, it is important to set the template 
width (along X direction) to not more than the interval 
between vehicles in order to enable a separating point to be 
recognized between vehicles. At the same time, the template 
length (width along Y direction) may be of any size if in a 55 
range processable by correlation calculation (such a size as 
to permit grasping the features of the vehicle). 

5. Method of template registration 

A method of registering a template will be explained. A 
method of extracting a template followed includes using an 60 
image of a predetermined area (the simplest method being 
by using templates arranged on the vehicle-incoming side as 
shown in FIG. 16) always as a template. According to this 
method, however, a vehicle cannot be followed if it reap- 
pears after being hidden behind a large vehicle or the like. 65 
In view of this, a shade image of input associated with a 
unique point of the whole image, such as the one with a large 



differentiation value (spatial frequency feature) or the one 
with a large difference between images obtained by differ- 
entiations taken at predetermined time intervals is set as a 
template, and each template thus set is followed by the 
correlation calculation. These templates may be combined 
with images on the vehicle-incoming side. 

As another alternative, a learning processing is effected in 
such a manner as to extract an object directly from an image 
by use of a neural network, and a candidate image of a 
moving object is directly extracted by use of the result of 
learning, so that a shade image of an input corresponding to 
the particular area is extracted as a template. 

By sequentially executing this processing, even if a 
vehicle appears in the midst of processing, the following of 
vehicles becomes possible at and after the time of appear- 

6. Method of correlation calculation 

Although the above-mentioned "partial correlation inte- 
gration method" is for executing the two-dimensional cor- 
relation calculation, the "method of correlation integration 
by concentration projection" involving a simplified process- 
ing has a similar effect. As an outline is shown in FIGS. 19A 
and 19B, the correlation calculation is effected on the basis 
of a one-dimensional concentration distribution for deter- 
mining a concentration projection (concentration accumu- 
lation) along ordinate for each lane. If a partial pattern 
within a predetermined width of this projection distribution 
is followed as a template, the amount of movement along 
ordinate is determined. The change in abscissa, however, 
cannot be detected. If the detection along abscissa is to be 
executed, the projection (concentration accumulation) along 
abscissa of the concentration in the vicinity of a matching 
point along ordinate is determined when the particular 
matching point is obtained as shown in FIGS. 20A and 20B, 
and the correlation with a template in registration is defined. 
In this way, the change along ordinate is determined. The 
general procedure for the above-mentioned method is the 
same as the "method of partial correlation integration". 
Templates may be registered by a method placing sole 
emphasis on the change points of the projection distribution. 
The advantage of this method lies in that the processing time 
is reduced considerably as compared with the "partial cor- 
relation integration method" described above due to the fact 
that the template is one-dimensional. 

Although the above-mentioned methods refer only to the 
correlation calculation of a shade image, the binary template 
matching or hierarchical pattern matching (both shade and 
binary) which have so far been employed may be executed. 
In such a case, too, it is important to utilize the size, 
separation/integration of templates or updating of template 
patterns. 

The "method of partial correlation integration" and 
"method of partial correlation integration by concentration 
projection" according to the present invention can be used 
for wider applications. Apart from the human being or 
vehicles as an object, these methods are applicable also to 
the operation of following other types of moving objects. 
Some applications include: 

(1) Detection of stationary vehicles: By following vehicle 
traces, the position where a vehicle has stopped is 
determined. The illegal parking, etc. is thus detectable. 

(2) Management of parking lot: A parking lot is monitored 
from a high point to recognize an unoccupied area by 
following vehicle traces. 

(3) Monitoring crossings: Whether any vehicle stays 
within the crossing range is recognized. The vehicle 
behaviour is a simple movement along lateral direction 
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(or along vertical direction depending on the camera 
position), and therefore vehicles can be followed in 
simple fashion. 
(4) Monitoring intruders (security): Whether any suspi- 
cious person has intruded during the night time, for 5 
instance, is recognized. Since a human being is an 
object of recognition, a partial template smaller than 
the width of the human being should be used. The 
detection of an intruder into the railroad track from the 
station platform can also be processed. 10 
According to the present embodiment, the behaviour of an 
object is accurately detectable by the partial correlation 
integration method even in the case of low contrast or 
superposed objects. This compares with the conventional 
methods in which an input image is binary-coded or a 15 
threshold value is set for a binary projection distribution, 
with the result that tuning is difficult with the processing 
performance subject to a considerable variation depending 
on such conditions as environments including brightness 
and shadow. 20 

The present embodiment, which lacks binarized threshold 
values, eliminates the tuning, and is applicable to environ- 
mental changes in versatile fashion. Also, moving objects 
can be followed by shade pattern matching by the updating 
of shade templates even in the case where the shapes of 25 
templates stored and the moving objects to be followed 
undergo a change. The invention is thus easily realizable for 
recognition of vehicles turning to the right at crossings or the 
behaviour of human beings. In addition, templates are traced 
as partial templates, and separated and integrated in post- 30 
processing, so that objects of varying sizes can be easily 
recognized. 

Another embodiment of the present invention will be 
described below with reference to the accompanying draw- 
ings. 35 

FIG. 21 is a block diagram showing a configuration of a 
vehicle recognition system according to another embodi- 
ment of the present invention. The image of a road taken by 
a TV camera 1 or the like is applied to an image memory 3 
through an A/D converter for converting an analog into a 40 
digital data. A shade image memory of about 8 bits (256 
gradations) is used as the image memory 3 according to the 
present embodiment. This vehicle recognition system 
includes a shade template memory circuit 4 for storing shade 
templates of shade images (8 bits and 256 gradations) of a 45 
vehicle. The shade image memory 3 and the shade template 
memory circuit 4 are so configured as to scan the interior of 
an memory by an address processor 5. The shade template 
memory circuit 4 has a plurality of shade templates which 
have stored therein shade images of various vehicles taken 50 
from various angles. The shade pattern matching circuit 6 is 
for matching between the image data 30 of the image 
memory 3 and the image memory 40 of the shade template 
memory circuit 4, and executes the normalized correlation 
calculation as shown by Equation (1). This embodiment is 55 
configured with the template cut-out circuit 8, the trace 
management table 9 and the separation/integration circuit 10 
eliminated from the system shown in FIG. 1. 

Any method of correlation processing having a similar 
advantage may be employed with equal effect. 60 

The operation of the image memory 3, the scanning and 
shade pattern matching circuit 4 and the operation of the 
shade pattern matching circuit 6 are controlled by the CPU 
7. More specifically, the address processor 5 and the shade 
pattern matching circuit 6 are activated by the CPU 7, an 65 
image address of Equation (1) is generated at the address 
processor 5, the data on the related address of the shade 
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template memory circuit 4 and the image memory 3 is read 
out, the data required for similarity calculation is determined 
at the shade pattern matching circuit 6, and the similarity r(u, 
v) is calculated at the CPU 7. 

The shade pattern matching circuit 6 will be briefly 
explained. The data relating to the shade template is capable 
of being calculated in advance. At the time of similarity 
calculation, therefore, the data relating to the input image is 
calculated and transmitted to the CPU 7. The data relating to 
the input image, in the case of Equation (1), is obtained at 
the shade pattern matching circuit 6 by the calculation of 
Equations (2) to (4). 

A diagram for explaining the shade pattern matching 
against an input image is shown in FIGS. 22A and 22B. 
Assume that four vehicles 13 are in the view as an object of 
matching as shown in FIG. 22A. Shade templates represent- 
ing the color and size (shape) of the vehicles are required to 
be prepared. While scanning with the pattern matching 
effected with a shade template T21 as shown in FIG. 22B, 
for example, a coordinate associated with an increased 
similarity r(u, v) is searched for. A point PI is determined as 
shown in FIG. 23A. In similar manner, the scanning with 
shade templates T22 to T24 makes it possible to determine 
points P2 to P4 as shown in FIG. 23A. As a result, once the 
coordinates of vehicle position at time point t and at time 
point t+At are determined as shown in FIG. 23B, the 
instantaneous moving speed and direction of each vehicle 
can be calculated. 

The advantage of the afore-mentioned normalized corre- 
lation processing is that to the extent that the pattern and the 
image of shade templates in registration are similar, a 
similarity is obtained to some degree even when brightness 
changes or a vehicle is hidden behind something (as in the 
case of vehicles superposed or hidden partially in the 
shadow of a building). More specifically, even a vehicle 
bearing a color similar to the brightness of the road surface 
can be recognized as a "vehicle" from the shape thereof. In 
conventional systems, the binary-coding operation at a cer- 
tain threshold value makes it very difficult to recognize a 
vehicle of low contrast. The method according to the present 
embodiment, by contrast, permits easy extraction of a 
vehicle. 

In recognizing the behaviour of a vehicle running along 
the road, full scanning for vehicle search as shown in FIG. 
22B is not always required. More specifically, in view of the 
fact that the moving range of a vehicle is limited to some 
degree, the next search range can be specified from the 
preceding behaviour of the vehicle. The limitation of the 
search range is executed in the same way as in the embodi- 
ment described above with reference to FIGS. 8 to 10A, 10B 
and Equations 5 to 8 and will not be explained again. 

Now, explanation will be made with reference to FIGS. 
24A to 24D about an example in which a vehicle is followed 
actually. 

FIG. 24A shows the case in which vehicles move upward 
in the screen. Vehicles are required to be followed sequen- 
tially by the images thereof entering the screen. For this 
purpose, a detection area is set at the lower part of the screen, 
and vehicle images entering this portion are followed 
upward. Several shade templates are prepared. First, a 
vehicle is detected in a search area. More specifically, the 
similarity processing is executed in the search area for each 
shade template. Assuming that vehicles are detected at 
coordinates PI and P2 as shown in FIG. 24B at time point 
t as a consequence, a following table is prepared as Table 1, 
in which the coordinates PI and P2 are registered at Xold 
and Yold, respectively. 
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At the same time, the shade template number Tno and the 
time Timeold are registered. Since the initial states of j 
following are involved, Xold, Yold and Timeold are regis- 
tered at XI, Yl and Timel respectively, and "0" at X2, Y2 
and Time2. XI, Y2 and Timel represent the preceding 
coordinate and time, and X2, Y2 and Time2 represent the 
present coordinate and time for following vehicles. Further, 2 
due to the initial stages, the preceding speed and direction of 
movement are not determined. As the next search area, 
therefore, the initial moving speed of "0" to maximum speed 
(150 km/h on an expressway, for example) and the initial 
change amount of the direction of movement of, say, 30 2 
degrees, are set, with a moving range designated (xs, ys, xe, 
ye are calculated with Vminat 0 km/h, Vmax at 150 km/h, 
9 at zero degree and § at 30 degrees). An image at time point 
tl is inputted, and the similarity is calculated with a shade 
template with Tno of "3" with respect to the search area for 3 
vehicle PI. As shown in Table 2, the point PI' of m 
similarity is substituted into the coordinate (X2, Y2). 



e the fol- 



This management makes it possible 
lowing of vehicles with ease. 

FIGS. 25, 26 and 27 are flowcharts showing specific steps 
of processing, in which FIG. 25 shows a main flow. 

(1) First, shade templates are registered and the manage- 
ment table is initialized. 

(2) An image is inputted, and vehicles are searched for in 
a detection area with respect to the particular image, 
thereby executing the following of vehicles. 

The vehicle search in the detection area is executed as 
follows, as shown in FIG. 26: 

(1) The shade template number is initialized, and the 
matching processing executed by the template number 

(2) If there is any coordinate with a similarity degree more 
than a threshold value, the maximum coordinate for 
such a similarity degree is determined, and the coor- 
dinate, the shade template number and the time are 



TABLE 2 



nag Xold Yold 



Similar processing is executed also for the vehicle at point 
P2. In the second and subsequent executions, the moving, 
speed can be determined. The moving speed V thus deter- 
mined and the direction 6 are used to determine and store the 
next search area by the method mentioned above. This is 50 
also the case with the processing at time point t2, which is 
expressed as in Table 3. 



registered as Xol, Yold, Tno and Timeold respectively 
in the following (tracking) table. 

(3) In the process, values of Xold, Yold and Timeold are 
stored in XI, Yl and Timel, respectively. 

(4) Then, a flag of the table is set and the next search area 
(initial value) is determined. 



TABLE 3 

Conditions at time t2 

nag Xold Yold Tno Timeold xl Yl Timel X2 Y2 Time2 a 
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The above-mentioned processing is executed also for all 
the shade templates. The vehicle detection process is thus 
complete. 

The processing for following vehicles is executed as 
shown in FIG. 27. 5 

(1) The counter of the following tabic is initialized. The 
pattern matching is effected by the shade template 
(Tno) with respect to the next table if the flag in the 
following table i is "0", and with respect to the search 
area if the flag is "1". 10 

(2) If there is any similarity degree more than a threshold 
value, the coordinate of maximum similarity degree is 
extracted. The coordinate is stored at X2, Y2 of the 
following table, the prevailing time at Time2. 

(3) A search area for the next image is calculated from the 15 
moving speed and direction determined from XI, Yl, 
X2, Y2, and the search area is stored. 

(4) After that, X2, Y2 and Time2 are copied at XI, Yl and 
Timel respectively. 20 

The afore-mentioned operation is performed for all the 
vehicles with a set flag, thereby completing the vehicle- 
following processing. 

Provision of a number of shade templates for detection of 
incoming vehicles would consume a very long time in ^ 
vehicle search at a detection area in FIGS. 24A to 24D, with 
the result that moving objects, if high in speed, could not be 
detected one by one. This problem may be solved by either 
of the two counlermeasurcs described below. 

(1) To reduce the shade templates as far as possible. 30 

(2) To prepare shade templates for following vehicles 
from an input image. 

The method (1) will be explained. Shade templates of 
only "white/small", "black/small", "white/large" and 
"black/large" used for vehicles are insufficient in number. In 35 
the case where a white/small template is available, for 
instance, the vehicles, though all small in size, cannot be 
detected by the matching with patterns of different shapes 
due to variations including sedan, van, jeep and vehicle with 
loading space. It is, however, very time-consuming and 40 
unrealistic to store all types of shade templates for matching. 

In view of this, a method of preparing shade templates 
will be explained with reference to FIGS. 28 to 30. Assume 
that there are eight shade templates T(l) to T(8) prepared as 
shown in FIG. 28. The similarity degree of each template in 45 
comparison with the others is checked by matching, say, 
template T(l) with the other shade templates. If there is any 
pair of similar templates, both the templates of the particular 
pair are used to prepare a new pattern of template, while the 
original one is discarded thereby to reduce the number of 50 
templates. In the case where pattern T(l) is similar to pattern 
T(6), and pattern T(3) to pattern T(5), for example, the 
images of both patterns T(l) and T(6) are averaged to 
prepare a pattern T'(l). Also, a pattern T'(3) is prepared by 
averaging the images of patterns T(3) and T(5). 55 

As a result of combining images by determining the 
similarity once in the manner mentioned above, the shade 
templates are reduced to six as shown in FIG. 29. By similar 
matching between the patterns shown in FIG. 29 and the 
initial shade templates T(l) to T(8) shown in FIG. 28, new 60 
shade templates shown in FIG. 30 are synthesized, thereby 
reducing the total number of templates. If this process is 
repeated until the similarity degree larger than a certain 
threshold value is eliminated, the shade templates finally 
obtained meets the necessary minimum for extracting 65 
vehicles. The processing time is thus remarkably reduced. 
Instead of preparing a new shade template by averaging 
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images as mentioned above, the maximum or minimum 
concentration of two images may be reserved. 

Now, the method (2) will be explained. According to the 
method (2), shade templates are not prepared in advance. 
Instead, images to be followed are automatically cut out of 
an input image taken by way of TV camera, and are used as 
shade templates. In this case, as many templates as vehicles 
to be followed suffices. 

This method will be explained with reference to FIGS. 31 
to 33. FIG. 31 shows a case in which a vehicle is moving 
upward in the screen (the dashed line defines lane areas). In 
order to follow this vehicle, an area sufficient covering the 
vehicle is cut out of the image at the lower part of the screen, 
and is used as a shade template as shown in FIG. 31. If an 
image not covering the vehicle at the center of the area to be 
cut out is registered as a shade template as shown FIG. 32, 
the subsequent following process is adversely affected. The 
concentration of the image thus cut out is checked for any 
deviation to periphery, and if need be, the position and size 
of the area to be cut out are changed. 

Taking note of the fact that the vehicle pattern is laterally 
symmetric, the processing mentioned below is executed. 

(1) The cumulative distribution of concentration along 
vertical direction of the cut-out image is determined 
(cumulative distribution of concentration along 
abscissa). 

(2) The center of gravity xg and the variance ax of the 
concentration of the distribution thus obtained is deter- 

(3) In similar fashion, the cumulative distribution of 
concentration is determined along horizontal direction 
of the image cut out (cumulative distribution of con- 
centration along ordinate). 

(4) The center of gravity yg and the variance cry of 
concentration of the distribution thus obtained are 
determined. 

(5) With the concentration center of gravity xg, yg as a 
central coordinate, an area of the size given as 

is cut out, where a is a constant (FIG. 33). 
(6) The processing mentioned above is executed again for 
the cut-out area and is repeated until the center-of-gravity 
coordinate and variance come to remain unchanged. 

The aforementioned processing makes it possible to reg- 
ister a shade template of a minimum area (FIG. 34) sur- 
rounding a vehicle. In the case where the concentration is 
sided in the end, the vehicle is assumed to be not covered 
completely in the screen, and the registration of the shade 
template is suspended (and reprocessed with an image 
inputted at the next time point). 

The embodiments described above refer to the processing 
with a road viewed from right above. In actually taking a 
picture of a road, however, it is virtually impossible to 
photograph a road from right above. The inevitable result is 
taking a bird's-eye view of a road. In such a case, the road 
presents such a view that a vehicle, with the movement from 
Tl to T2 starting with time TO, appears the smaller the 
farther it runs away, as shown in FIG. 35A. If a shade 
template is followed in the manner explained with reference 
to FIG. 31, a vehicle appearing to be of the same size as the 
shade template could be detected but a far vehicle could not 
be detected and finally fails to be followed.' 

This inconvenience is overcome by updating the shade 
template while following the vehicle as shown in FIG. 35B. 
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More specifically, when a point PO is detected by a shade 
template, another shade template for the next search is 
prepared by use of an image proximate to the point PO. In 
this method, as in the method described above, the concen- 
tration center of gravity and variance may be used. The 5 
shade template thus newly registered is used for matching 
processing for the next search area thereby to determine the 
point PI. A shade template is prepared by using an image in 
the vicinity of the point PI. This process is sequentially 
executed, so that vehicles, even if changed in size, can be to 
followed in versatile fashion. This method is based on the 
fact that the vehicle size rarely changes instantaneously. 

A template may alternatively be prepared by linear cou- 
pling operation (ctf+p*g, a +(i=l) or by determining an 
average ((f+g)/2) of the image (f) in the vicinity of a 15 
newly-detected vehicle position and the preceding template 
image (g). 

If this technique is used, a vehicle can be followed easily 
by updating the shade template at the same time, unlike 
when using a single template in which case the following of 20 
a vehicle is suspended midway in measuring the number of 
vehicles turning to the right at an intersection, in view of the 
fact that the vehicle turning to the right changes its direction 
slowly, as shown in FIG. 12. An alternative method consists 
in reducing the size of the shade template according to the 25 
vehicle size, i.e., according to the ratio determined by the 
ordinate of the vehicle position determined. In this method, 
however, the change only in size can be dealt with. 

According to the present embodiment, as described 
above, the behaviour of vehicles, even if low in contrast or 30 
superposed, can be detected accurately by the shade pattern 
matching. Also, the updating of a shade template makes it 
possible to execute the following of a moving object by 
pattern matching even when the shape of the template stored 
or the moving object undergoes a change. Especially, easy 35 
recognition of a vehicle turning to the right at an intersection 
or the like is possible. 

This embodiment is also applicable to the detection of a 
stationary vehicle (such as illegally parked) or management 
of a parking lot (detection of an occupied section). 40 

Another embodiment of the present invention as applied 
to an abnormality detection system for a road, etc. will be 
explained. 

A general configuration of the present embodiment is 
shown in FIG. 36. An abnormality detection system 200 45 
includes a plurality of image processing units 100-1, 100-2 
and 100-3 and a central overall determination section 22. In 
order to grasp various abnormal phenomena, each of image 
processing units 100-1, 100-2 and 100-3 includes a traffic 
flow measuring section 26 for collecting various information 50 
by processing input images of objects, an abnormal run 
monitor 14, a stationary vehicle detector 16, a congestion 
degree measuring section 18 and a local overall determina- 
tion section 19 (19-1, 19-2, 19-3). The central overall 
determination section 22, on the other hand, is for collecting 55 
data on each image processing unit and making an overall 
determination by spatial interpolation, for example. 

The operation of each section will be briefly explained. 
The image processing unit 100 is inputted with an image 
(section 23), extracts a background difference feature with 60 
respect to the image (section 24), follows vehicles (section 
25), and measures the resulting traffic flow (number and 
speed of vehicles) (section 26). Also, the frame feature of an 
input image is extracted (section IS) and a stationary vehicle 
is detected (section 16). In addition, features of spatial and 65 
temporal differentiations with respect to an input image are 
extracted (section 17) and the degree of congestion is 
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measured (section 18). An overall decision is made on these 
results at the local overall determination section 19 (20, 21) 
to determine the presence or absence of any abnormal 
phenomenon. At the same time as the abnormality determi- 
nation at each unit, all the measurement information such as 
congestion degree and speed obtained at each unit are 
collected by the central overall determination section 22, 
and after spatial interpolation, an abnormal phenomenon 
outside of the visual field of the camera is detected. 
Each section will be described below in detail. 

(1) Traffic flow measurement section 

The traffic flow measurement data includes the vehicle 
speed and the number of vehicles passed. An example of the 
method for obtaining these data is shown in FIG. 37. The 
steps for processing are described below. 

The background image 30 stored in advance is removed 
from the input image f(t) 31 at time point t thereby to extract 
a vehicle. 

The end coordinate of the vehicle image 32 extracted (as 
in the case where the vehicle is photographed from behind) 
is detected (33). 

The above-mentioned steps 35 and 36 are executed with 
the image f(t+dt) sequentially obtained, and the position to 
which the end coordinate has moved is determined by the 
coordinate-following operation 37. 

The number of vehicles passed 38 and the speed 39 are 
measured from the coordinate change followed. 

The vehicle flow within the visual field of the camera can 
be measured by this processing. It is possible to determine 
by monitoring this data that an abnormality has occurred, for 
example, when the vehicle speed has abnormally decreased. 

When the end coordinate of a vehicle is extracted, the 
range of vehicle existence, i.e., the coordinates at the 
extreme right and left ends of the vehicle are measured at the 
same time in order to monitor for an abnormal run as 
described below. 

(2) Abnormal run monitor section 

A small fallen object or the like cannot be easily detected 
by the function of stationary vehicle detection described 
below. In the presence of a fallen object, however, the 
vehicle is normally driven avoiding the object, and therefore 
an abnormal running pattern is generated. This function is 
for monitoring such an abnormal running pattern. Many 
vehicles running in abnormal manner are liable to straddle 
the lane boundary, frequently change lanes or change the 
speed suddenly. 

In order to detect such phenomena as mentioned above, 
the present invention uses the processing as shown in FIG. 
38. A road image is designated as 40. 

A vehicle is extracted from an input image as in the traffic 

The various coordinates of the vehicle are determined 
from the vehicle image extracted. They include the distance 
ds to the coordinate at the extreme left end of the vehicle, the 
distance de to the coordinate at the extreme right end of the 
vehicle and the central coordinate. 

Each vehicle is followed at the time of traffic flow 
measurement, and the change of the coordinates thereof is 
determined. As a result, the instantaneous speed and direc- 
tion of movement are obtained. 

After the above-mentioned processings, the respective 
data are reserved as an accumulated data as shown in FIGS. 
39A to 39C for abnormality determination. 

Accumulated data on the coordinates at the extreme left 
and right ends provide information to determine a range in 
which vehicles are running frequently within a lane as 
shown in FIG. 39A. It is thus possible to determine that a 
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vehicle is running abnormally when the frequency of dis- 
placement from a normal drive range exceeds a predeter- 
mined threshold value. 

Determination can also be made on whether the data on 
speed or direction of movement, if managed in similar 5 
fashion as shown in FIGS. 39B and 39C, frequently assumes 
an abnormal value. 

(3) Function of stationary vehicle detection 

In the case where there is any stationary vehicle present 
within the visual field of the camera, direct detection of the 1Q 
stationary object is more desirable than indirect measure- 
ment like the monitoring of an abnormal run. More specifi- 
cally, a stationary object detectable by imaging (a stationary 
vehicle, a somewhat large fallen object, etc.) is detected 
directly. 

A method of detecting a stationary object is shown in FIG. 15 
40. An outline of this processing is described below. 

An image is inputted at predetermined time intervals dt as 
indicated by f(t— dt) 42 and f(t) 43, and a background image 
41 stored in advance is removed from each input image. 

The images 44, 45 from which the background has been 20 
removed have only a vehicle image remaining therein. A 
moving object is detected by use of these images. A method 
of detecting a moving object is by determining the speed as 
in the measurement of traffic flow. In the case under con- 
sideration, however, a moving object is detected by the 25 
features between image frames. More specifically, two 
images are subjected to a differentiation processing to 
extract a moving area (46). 

The area from which a moving area image 46 is removed 
from an image lacking the background makes up a candidate 30 
image of a stationary object (47). Upon confirmation that the 
position of this object remains stationary, it is determined 
that there exists a stationary object as a final result (48). 

A stationary object or a fallen object can be detected 
within the visual field of the camera by executing the 35 
above-mentioned processing. 

(4) Function of congestion measurement 

An abnormal phenomenon outside of the visual field of 
the TV camera is impossible to detect directly. In view of 
this, a phenomenon as a repercussion of an abnormality 40 
occurring outside of the camera which enters the visual field 
thereof, i.e., the congestion degree, is measured. In the case 
where the congestion degree assumes a value different from 
a normal one, it may be decided that some abnormality has 
occurred in the direction forward of the vehicle. 45 

The congestion (traffic jam) degree may be measured by 
various methods. The method employed in the present 
invention uses no traffic flow data but is based on the 
macroscopic measurement of the number and speed of 
vehicles taken by such expressions as "vehicles are many", 50 
"vehicles are few", "vehicle speed is high" or "vehicle speed 
is low" as shown in FIG. 41. This is due to the fact that with 
the intensification of vehicle congestion, the images taken 
by the TV camera are superposed one on another, thereby 
making impossible the microscopic measurement of the 55 
number and speed of vehicles. Especially, vehicles appear 
superposed very often in a tunnel where TV cameras are 
installed at low level. 

The processing will be described briefly below. 

An input image f(t) 50 is differentiated (with a profile 60 
perpendicular to the running direction extracted) (spatial 
differentiation 51), and the feature amount relating to the 
vehicle quantity is calculated (52). In the case under con- 
sideration, a differentiated image is binary-coded and the 
number of vehicle profiles is determined. 65 

Further, an image f(t+dt) 53 is inputted at predetermined 
time intervals dt (say, 200 ms), and a difference image is 



22 

determined for each picture element of f(t) 50 and f(t+dt) 53 
(temporal differentiation 54). If a vehicle is moving, a 
differentiation image in some form or other appears. In the 
case where the vehicle remains stationary, on the other hand, 
no information is obtained. The feature amount (such as 
width) of an image obtained from the differentiation image 
is therefore determined as a data relating to speed (55). 

The feature amounts relating to the vehicle quantity and 
speed mentioned above are applied to a decision function 56 
engaged in learning in advance for calculation of the con- 
gestion degree. A neural network is used for the decision 
function under consideration. 

An abnormal phenomenon can be monitored by the 
feature amount like the vehicle quantity or speed as well as 
by the congestion degree determined from the above-men- 
tioned processing. 

(5) Local overall determination section 

The processing by the local overall determination section 
is executed mainly for improving the reliability of the 
information obtained from the functions of" "traffic flow 
measurement", "measurement of abnormal run", "detection 
of stationary vehicles" and "congestion degree measure- 
ment". Abnormal phenomena include local ones such as 
stationary vehicles (accidents) and overspeed, and abnor- 
malities covering wide areas like traffic congestion. An 
alarm should be issued against a local abnormality at the 
time point of detection by each image processing unit. 
Nevertheless, the problem of measurement accuracy makes 
it necessary to check the measurement result. Also, the 
manner in which an alarm is issued against a congestion 
apparently covering a wide range depends on whether the 
particular congestion is confined within the visual field of 
the camera or covers all the measurement points. As a result, 
information on an abnormality covering a wide range (con- 
gestion degree, measurement result of traffic flow, etc.) is 
sent to a host system to make an overall decision (determi- 
nation). 

The information applied to the local overall determination 
section includes: 
Traffic flow: Speed is high or low, traffic volume large or 

Abnormal run: Frequent or infrequent overspeed, fre- 
quently or infrequently displaced from normal running 
course 

Stationary vehicle: Presence or absence 

Congestion degree: high, middle or low 

Of these information, local abnormalities relate to 
whether a case of overspeed or stationary vehicle is detected 
or not. As to the overspeed, a contradictory data indicating 
an overspeed at the time of congestion measured by the 
abnormal run function is considered impossible and is 
cancelled. The relationship between the measurement result 
of the congestion measurement function and the speed 
information is thus checked and a contradictory data is 
cancelled to prevent an overalarm. The information obtained 
in detecting a stationary vehicle, however, is reliable and 
may be used to issue an alarm immediately upon detection. 

If the degree of an error developed by the speed data 
measured in accordance with the congestion level is grasped 
in advance, on the other hand, the speed data can be 
corrected in accordance with the congestion degree deter- 
mined. This is also the case with the number of vehicles. 

The local overall decision section checks for a data 
contradiction, and transmits information to a central deter- 
mination section 22 as a host system, while at the same time 
controlling the information on the presence of a stationary 
vehicle or an abnormal run as determined under a condition 
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allowing accurate speed measurement ("unbusy state") in 
such a manner as to permit an immediate alarm output. 
(6) Central overall determination section 

In the event that TV cameras are installed at intervals of 
200 m, for example, the range actually measurable by 5 
imaging covers only about one half of this distance. An 
abnormal phenomenon occurring at a position not monitored 
by imaging, therefore, is required to be subjected to inter- 
polation (estimation) from the result of processing of adja- 
cent TV cameras. 1Q 

An interpolation processing described below is effected 
according to the present invention. 

(i) Spatial interpolation 

The spatial interpolation at the central overall determina- 
tion section is to spatially grasp and interpolate the result of l5 
processing the image of each TV camera. The spatial inter- 
polation enables a determination on whether a given con- 
gestion is a chronic one or due to an accident or a stationary 
vehicle, or an estimation of any abnormality in an area not 
covered by the camera. For example, the congestion degree 2 o 
at each ground point is plotted two-dimensionally as shown 
in FIG. 42, and it is decided that some abnormality has 
occurred at or in the vicinity of a point where the congestion 
degree has considerably changed as compared with the 
neighbouring conditions. In FIG. 42, the congestion degree 25 
of the left lane is low, while that of the opposite lane (passing 
lane) is high. It is, therefore, estimated that abnormally many 
vehicles change lanes in this section. 

More specifically, the congestion degree obtained from 
each image processing unit is studied in overall fashion. It 30 
is decided that an abnormality is caused outside of the 
measurement section if traffic is congested in all sections, or 
that the congestion is a chronic one if the traffic congestion 
is limited to part of the measurement range, so that an 
abnormality decision is made only when traffic is congested 35 
only in part of the measurement range. 

All the information acquired are processed as data, and 
any part where the congestion degree is locally different is 
searched for. If there is found any such part, an abnormality 
decision is made. Otherwise, a decision is made that the 40 
traffic is normal. The data is analyzed either by a general 
waveform (one-dimensional data on congestion degree with 
respect to ground point) analysis technique or by a method 
in which a waveform is directly inputted to a neural network 
to detect a change point. 45 

Objects of data analysis may include not only the con- 
gestion degree but also the data on abnormally running 
vehicles monitored (lane data, speed data, etc.). It is possible 
to make an abnormality decision when vehicles are dis- 
placed from a normal course only at a part of the range. 50 

In general, an attendant observes the monitor screen to 
check for any abnormal phenomenon. If a spatial graph as 
shown in FIG. 42 is indicated to present the result of 
automatic decision to the monitor attendant, the conditions 
of the entire measurement range become easy to grasp. 55 

An abnormality is an invisible phenomenon, and therefore 
the type thereof cannot be specified, although a decision as 
to the presence or absence of an abnormality is possible. The 
coordinate distribution, speed distribution, etc. of vehicles, 
instead of the congestion degree plotted in FIG. 42, may be 60 
spatially determined. 

(ii) Temporal interpolation 

The data measured at each image processing unit is 
collected and managed in time series as shown in FIG. 43. 
A decision is made on the time of occurrence or on whether 65 
the abnormalities are of primary nature (such as the effect of 
a noise specific to the image processing) or of sustaining 
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nature. This decision is made by waveform analysis as in the 
case of the spatial interpolation described above. 

As a result, a noise-related abnormality is removed for an 
improved system reliability. Also, the indication to the 
monitor attendant facilitates the grasping of a chronological 
change as in the case of spatial interpolation. 

An example of hardware configuration for realizing the 
abnormal phenomenon detection mentioned above is shown 
in FIG. 44. Video images from a plurality of TV cameras 60 
installed appropriately are applied to image processing units 
100, which execute "traffic flow measurement", "monitoring 
of abnormally-running vehicles", "detection of stationary 
vehicles" and "congestion degree measurement", and trans- 
mit the result of the executions to a central overall deter- 
mination section 22 as a host system. 

Each of the image processing units 100 includes an A/D 
converter 70 for converting a video signal of the TV camera 
60 into a digital signal, an image memory 71 for storing the 
resulting data, an image processor 72 for processing the data 
of the image memory 71, a data output section 73 for 
transmitting the data to the central overall determination 
section 22, and a CPU 74 for controlling these devices, as 
shown in FIG. 45. The image processor 72 is capable of 
executing the inter-image operations such as subtraction and 
addition of images or a spatial sum-of the products operation 
such as differentiation, binary coding or histogram process- 
ing. The local overall decision is processed at the CPU 74. 

The central overall determination section 22, as shown in 
FIG. 46, includes an interface section 74 for transmitting and 
receiving a data with each image processing unit 100 not 
shown, a data accumulator 75 for reserving the data thus 
obtained, and a data analyzer 76 for subjecting the accumu- 
lated data to spatial and time interpolation. 

According to the present embodiment, as described 
above, various abnormal phenomena are capable of being 
detected by the functions of "traffic flow measurement", 
"detection of abnormally-running vehicles", "detection of 
stationary vehicles", "congestion degree measurement" and 
"inter-camera interpolation". 

In addition to the conditions in the tunnel, as described 
above, the road conditions in general can of course be 
monitored with equal effect according to the present inven- 

We claim: 

1. An object recognition system for photographing an 
object by a TV camera and recognizing movement of the 
object by processing images from the TV camera, compris- 
ing: 

a template extracting circuit extracting partial templates 

from an image of an object; 
a following means for following the object by correlation 

calculation between the partial templates and an input 

image; and 

means for separating/integrating the partial templates on 
a basis of a selected one of a history of coordinates 
followed and a knowledge relating to the object. 

2. An object recognition system according to claim 1, 
wherein said template extracting circuit extracts images of a 
plurality of partial areas formed on an object-incoming side 
of an input image as templates. 

3. An object recognition system according to claim 1, 
wherein said template extracting circuit extracts an input 
image corresponding to an area having selected one of a 
large spatial frequency feature and a large time frequency 
feature as a template of predetermined size. 

4. An object recognition system according to claim 1, 
wherein the template extracting circuit extracts an area 
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representing a similarity of a moving object from an input 
image and extracts an input image corresponding to a 
particular area as said templates. 

5. An object recognition system according to claim 1, 
wherein said separation/integration circuit separates or inte- 5 
grates templates having a similar speed and direction of 
movement of an object on a basis of the trace of movement 
determined for each template. 

6. An object recognition system according to claim 1, 
wherein said following means comprises: ^ 

a means for learning a past behavior pattern by a neural 
network; 

a means for predicting future behavior from the past 
behavior pattern and a present behavior pattern; and 

a means for determining a range of correlation calculation 1 3 
with respect to templates. 

7. An object recognition system according to claim 1, 
further comprising a template updating circuit for updating 
a template image stored by use of an input image. 

8. An object recognition system according to claim 7, 
wherein the template updating circuit matches patterns 
between an input image f(t) at time point t and a template g, 
and in a case where a coordinate P with a similarity degree 
larger than a predetermined threshold value is determined, 
an image in the vicinity of the coordinate P of input image 2f 
f(t) is registered newly as a gray scale template g'. 

9. An object recognition system according to claim 7, 
wherein the template updating circuit matches patterns 
between an input image f(t) at time point t and a template g, 
and in a case where a coordinate P with a similarity degree 3t 
larger than a predetermined threshold value is determined, a 
new template expressed by g'=af (t)+fig (a, p: constants) 
using an image f (t) in the vicinity of the coordinate P of all 
the input images f(t) is registered. 

10. An object recognition system for photographing an 3: 
object by a TV camera and recognizing movement of the 
object by processing input images from the TV camera, 
comprising: 

a template extracting circuit extracting a plurality of 4( 
images in a template area of predetermined size from 
an input image f(t) at a time point t; 

a template memory circuit storing a plurality of images 
obtained by the template extracting circuit; 

a pattern matching circuit determining a movement point At 
of each template by shade pattern matching between 
each template stored in the template memory circuit 
and the input images at incrementing time points 
obtained from the TV camera; 

a trace management table storing a relationship between 51 
a time point and a movement point for each gray scale 
template obtained at the gray scale pattern matching 
circuit; and 

a separation/integration circuit deciding which templates 
represent an image of a same object by use of the trace 5 ' 
management table. 

11. A system for measuring an intersection traffic flow 
comprising: 

a template extracting circuit extracting a plurality of & 
images of a template area of predetermined size from 
an image at time point t; 

a gray scale template memory circuit storing a plurality of 
templates obtained with the template extracting circuit; 

a pattern matching circuit matching patterns between each 6 
of the templates stored in said template memory circuit 
and input images at time points (t+dt), (t+2dt), . . . 
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(t+ndt) (n: integral number) obtained from a TV camera 
and determining a movement point of each of said 
templates; 

a template updating circuit sequentially updating the 
templates of said template memory circuit using an 
input image; 

a trace management table storing a relationship between 
a time point and a moving point for each template 
obtained with the pattern matching circuit; 

a separation/integration circuit deciding on which tem- 
plates represent a same object by use of the trace 
management table; and 

a traffic parameter measurement section measuring a 
number and speed of vehicles passed, a number of 
vehicles turned to a right/left by following a vehicle 
position obtained. 

12. A vehicle recognition system for photographing a road 
by a TV camera and recognizing movement of vehicles by 
processing images from«the TV camera, comprising: 

a template memory circuit for storing several types of 
vehicles images as images in advance; and 

a pattern matching circuit for determining a vehicle 
position within an input image by pattern matching 
between the templates stored in the template memory 
circuit and the input image obtained from the TV 

13. A vehicle recognition system according to claim 12, 
further comprising a template pattern preparation circuit 
having a plurality of template patterns T(0) to T(n) for 
determining a similarity degree between the template pat- 
terns and for preparing a new template pattern from patterns 
similar to each other. 

14. A vehicle recognition system fo 
by a TV camera and recognizing vehicle 11 
processing images from the TV camera, comprising: 

a template extracting circuit for extracting an image 
pattern corresponding to a vehicle as a template from 
an input image obtained by the TV camera; 

a template memory circuit for storing a template obtained 
with the template extracting circuit; 

a pattern matching circuit for determining a vehicle 
position within a subsequent input image by pattern 
matching between the template stored in the template 
memory circuit and said subsequent input image; and 

a template extracting circuit for cutting out an area of 
predetermined size from an image entered by a vehicle, 
determining an extracting position in such a manner as 
to locate the vehicle at a center thereof from features of 
an extracted image, and registering an input image of 
the extracting position and a vicinity thereof as tem- 



15. A traffic flow measurement system for photographing 
a road by a TV camera and recognizing vehicle m 
by processing images from the TV can 

a template memory circuit storing a vehicle image corre- 
sponding to a vehicle as a template 
a pattern matching circuit pattern matching between the 
template stored in the template memory circuit and a 
subsequent input image obtained from the TV camera; 
and 

a vehicle speed and direction determining means for 
determining a vehicle position by pattern matching 
with respect to the input image at a given time point, 
further determining the vehicle position with respect to 
the input image at a given time point and deterrnining 
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at least a speed and direction of movement of vehicles, 
in accordance with a change in coordinates of deter- 
mined positions; 
wherein said vehicle speed and direction determining 
means determines a range of pattern matching of the 5 
template against the input image f(t+At) at a time point 
t+At on a basis of the speed and direction of movement 
obtained from a result of pattern matching of images 
f(t-At), f(t) at time points (t-At) and t respectively. 

16. A pattern matching system for executing pattern 10 
matching between a template and an input image, compris- 
ing: 

a template memory circuit storing a pattern of an image; 

a pattern matching circuit pattern matching between a 
template stored in the template memory circuit and a 15 
subsequent input image obtained from a TV camera; 

a template updating circuit sequentially updating the 
templates of the template memory circuit using input 
images; 20 

wherein pattern matching is effected with a given tem- 
plate g against an input image f(t) at time point t, and 
in a case where a coordinate P is determined to have a 
similarity degree obtained by pattern matching larger 
than a predetermined threshold value, an input image 25 
f(t) of the coordinate P and a vicinity thereof is regis- 
tered as a new template g'. 

17. A pattern matching system for executing pattern 
matching between a template and an input image, compris- 
ing: 30 

a template memory circuit storing a pattern of an image; 
a pattern matching circuit pattern matching between a 

template stored in the template memory circuit and a 

subsequent input image obtained from a TV camera; 

and 35 
a template updating circuit sequentially updating the 

templates of the template memory circuit using input 

images; 

wherein pattern matching is effected with an input image 
f(t) at a time point t against a template g, and in a case 40 
where a coordinate P is determined to have a similarity 
degree larger than a predetermined threshold value, an 
input image f(t) of the coordinate P and a vicinity 
thereof is used to register a template expressed as 
g'=af (t)+pg (a, |5: constants) as a new template. 45 

18. A pattern matching system for executing pattern 
matching between a template and an input image, compris- 
ing: 

a template memory circuit storing a pattern of an image; 

a pattern matching circuit pattern matching between a 50 
template stored in the template memory circuit and a 
subsequent input image obtained from a TV camera; 

a template updating circuit sequentially updating the 
templates of the template memory circuit using input 55 
images; 

wherein a template is updated while changing a template 
size in accordance with a size of an object. 

19. A system for following, in a picked-up image, a 6Q 
moving object of a shape and size changing with the 
direction of movement and distance from the image pick-up 
means, comprising: 

a template memory circuit for storing a pattern of an 
image of the moving object as a template; 65 

a pattern matching circuit for determining a position of 
the moving object by pattern matching between the 



template stored in the template memory circuit and an 
input image obtained from the image pick-up means; 

a template updating circuit for preparing an updated 
template for a next pattern matching from the template 
of the moving object cut out of a present input image 
and a preceding template of the moving object and 
storing the updated template in the template memory 

20. An abnormality detection system for detecting an 
abnormal phenomenon by photographing a road with a 
plurality of TV cameras installed at predetermined intervals 
on the road and processing images from the TV cameras, 
comprising: 

a plurality of image processing units including a local 
overall decision section having functions of processing 
images from the TV cameras and measuring a speed 
and number of vehicles, monitoring abnormally-run- 
ning vehicles, detecting a stationary object and mea- 
suring a congestion degree, and correcting data on the 
speed and number of vehicles and the abnormally- 
running vehicle monitor functions in accordance with 
the congestion degree; and 

a central overall decision section making an overall 
decision by spatial and temporal interpolation of a 
result of measurement at the image processing units. 

21. An abnormality detection system according to claim 
20, wherein the central overall decision section includes a 
means for collecting several types of information from a 
plurality of image processing units, a spatial interpolation 
means for grasping said information spatially and making an 
abnormality decision only when said information undergoes 
a local change, and a time interpolation means for grasping 
a plurality of information in time series and deciding on an 
initial time of abnormality occurrence and an instantaneous 
abnormality. 

22. An abnormality detection system for detecting an 
abnormal phenomenon by photographing a road with a TV 
camera and by processing images from the TV camera, 
comprising: 

a local overall decision section having functions of mea- 
suring a speed and number of vehicles, monitoring 
abnormally-running vehicles, detecting a stationary 
object and measuring a congestion degree for correct- 
ing data on the speed and number of vehicles and 
abnormally-running vehicle monitor functions in 
accordance with the congestion degree; 

wherein the function of monitoring abnormally-running 
vehicles is performed using a means for extracting 
vehicles from an input image, a means for determining 
an extreme left coordinate, extreme right coordinate 
and end coordinate of vehicles from a vehicle image 
extracted, a means for calculating an amount of vehicle 
movement from coordinates thus determined and an 
image inputted at a next time point, a means for 
accumulating resulting coordinate values and speed 
and direction of movement for a number of vehicles 
involved, and means for making an abnormality deci- 
sion when coordinates exceed a threshold value of a 
frequency of vehicles running outside a predetermined 
position, a frequency of vehicles running at other than 
a predetermined speed and a frequency of vehicles 
moving in other than a predetermined direction, respec- 

23. An abnormality detection system for detecting an 
abnormal phenomenon by photographing a road with a TV 
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camera and by processing images from the TV camera, 
comprising: 

a local overall decision section having functions of mea- 
suring a speed and number of vehicles, monitoring 
abnormally-running vehicles, detecting a stationary 
object and measuring a congestion degree for correct- 
ing data on the speed and number of vehicles and 
abnormally-running vehicle monitor functions in 
accordance with the congestion degree; 

wherein the local overall decision section cancels a speed 
data obtained with a high congestion degree, when the 
speed data is higher than a predetermined threshold 

24. An abnormality detection system for detecting an 
abnormal phenomenon by photographing a road with a TV 
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camera and by processing images from the TV camera, 
comprising: 

a local overall decision section having functions of mea- 
suring a speed and number of vehicles, monitoring 
abnormally-running vehicles, detecting a stationary 
object and measuring a congestion degree for correct- 
ing data on the speed and number of vehicles and 
abnormally-running vehicle monitor functions in 
accordance with the congestion degree; 

wherein the local overall decision section corrects data on 
the speed and the number of vehicles determined, in 
accordance with the congestion degree. 
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MANUAL OF PATENT EXAMINING PROCEDURE 



reduced. An article published several years later by 
Knudsen opined that the Carlisle technique does not 
yield appreciable improvement in bubble oscillation 
suppression. However, the article did not test the Car- 
lisle technique under comparable conditions because 
Knudsen did not use Carlisle's spacing or seismic 
source. Furthermore, where the Knudsen model most 
closely approximated the patent technique there was a 
30% reduction of the secondary pressure pulse. On 
these facts, the court found that the Knudsen article 
would not have deterred one of ordinary skill in the 
art from using the Carlisle patent teachings.). 

III. FACT THAT REFERENCES CAN BE 
COMBINED OR MODIFIED **>MAY 
NOT BE< SUFFICIENT TO ESTABLISH 
PRIMA FACIE OBVIOUSNESS 

The mere fact that references can be combined or 
modified does not render the resultant combination 
obvious unless **>the results would have been pre- 
dictable to one of ordinary skill in the art. KSR Inter- 
national Co. v. Teleflex Inc., 550 U.S. , , 82 

USPQ2d 1385, 1396 (2007)("If a person of ordinary 
skill can implement a predictable variation, § 103 
likely bars its patentability. For the same reason, if a 
technique has been used to improve one device, and a 
person of ordinary skill in the art would recognize that 
it would improve similar devices in the same way, 
using the technique is obvious unless its actual appli- 
cation is beyond his or her skill. ").< 

IV. *>MERE STATEMENT< THAT THE 
CLAIMED INVENTION IS WITHIN THE 
CAPABILITIES OF ONE OF ORDINARY 
SKILL IN THE ART IS NOT SUFFICIENT 
BY ITSELF TO ESTABLISH PRIMA FA- 
CIE OBVIOUSNESS 

A statement that modifications of the prior art to 
meet the claimed invention would have been "' well 
within the ordinary skill of the art at the time the 
claimed invention was made'" because the references 
relied upon teach that all aspects of the claimed inven- 
tion were individually known in the art is not suffi- 
cient to establish a prima facie case of obviousness 
without some objective reason to combine the teach- 
ings of the references. Ex parte Levengood, 
28USPQ2d 1300 (Bd. Pat. App. & Inter. 1993). 
**'">[R]ejections on obviousness cannot be sus- 



tained by mere conclusory statements; instead, there 
must be some articulated reasoning with some ratio- 
nal underpinning to support the legal conclusion of 

obviousness.'" KSR, 550 U.S. at , 82 USPQ2d at 

1396 quoting In re Kahn, 441 F.3d 977, 988, 78 
USPQ2d 1329, 1336 (Fed. Cir. 2006).< 

V. THE PROPOSED MODIFICATION CAN- 
NOT RENDER THE PRIOR ART UNSAT- 
ISFACTORY FOR ITS INTENDED 
PURPOSE 

If proposed modification would render the prior art 
invention being modified unsatisfactory for its 
intended purpose, then there is no suggestion or moti- 
vation to make the proposed modification. In re Gor- 
don, 733 F.2d 900, 221 USPQ 1125 (Fed. Cir. 1984) 
(Claimed device was a blood filter assembly for use 
during medical procedures wherein both the inlet and 
outlet for the blood were located at the bottom end of 
the filter assembly, and wherein a gas vent was 
present at the top of the filter assembly. The prior art 
reference taught a liquid strainer for removing dirt 
and water from gasoline and other light oils wherein 
the inlet and outlet were at the top of the device, 
and wherein a pet-cock (stopcock) was located at the 
bottom of the device for periodically removing the 
collected dirt and water. The reference further taught 
that the separation is assisted by gravity. The Board 
concluded the claims were prima facie obvious, rea- 
soning that it would have been obvious to turn the ref- 
erence device upside down. The court reversed, 
finding that if the prior art device was turned upside 
down it would be inoperable for its intended purpose 
because the gasoline to be filtered would be trapped at 
the top, the water and heavier oils sought to be sepa- 
rated would flow out of the outlet instead of the puri- 
fied gasoline, and the screen would become clogged.). 

"Although statements limiting the function or capa- 
bility of a prior art device require fair consideration, 
simplicity of the prior art is rarely a characteristic that 
weighs against obviousness of a more complicated 
device with added function." In re Dance, 160 F.3d 
1339, 1344, 48 USPQ2d 1635, 1638 (Fed. Cir. 1998) 
(Court held that claimed catheter for removing 
obstruction in blood vessels would have been obvious 
in view of a first reference which taught all of the 
claimed elements except for a "means for recovering 
fluid and debris" in combination with a second refer- 
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ence describing a catheter including that means. The 
court agreed that the first reference, which stressed 
simplicity of structure and taught emulsification of 
the debris, did not teach away from the addition of a 
channel for the recovery of the debris.). 

VI. THE PROPOSED MODIFICATION CAN- 
NOT CHANGE THE PRINCIPLE OF OP- 
ERATION OF A REFERENCE 

If the proposed modification or combination of the 
prior art would change the principle of operation of 
the prior art invention being modified, then the teach- 
ings of the references are not sufficient to render the 
claims prima facie obvious. In re Ratti, 270 F.2d 8 1 0, 
123 USPQ 349 (CCPA 1959) (Claims were directed 
to an oil seal comprising a bore engaging portion with 
outwardly biased resilient spring fingers inserted in a 
resilient sealing member. The primary reference relied 
upon in a rejection based on a combination of refer- 
ences disclosed an oil seal wherein the bore engaging 
portion was reinforced by a cylindrical sheet metal 
casing. Patentee taught the device required rigidity for 
operation, whereas the claimed invention required 
resiliency. The court reversed the rejection holding 
the "suggested combination of references would 
require a substantial reconstruction and redesign of 
the elements shown in [the primary reference] as well 
as a change in the basic principle under which the 
[primary reference] construction was designed to 
operate." 270 F.2d at 813, 123 USPQ at 352.). 

2143.02 Reasonable Expectation of Suc- 
cess Is Required [R-6] 

>A rationale to support a conclusion that a claim 
would have been obvious is that all the claimed ele- 
ments were known in the prior art and one skilled in 
the art could have combined the elements as claimed 
by known methods with no change in their respective 
functions, and the combination would have yielded 
nothing more than predictable results to one of ordi- 
nary skill in the art. KSR International Co. v. Teleflex 

Inc., 550 U.S. , , 82 USPQ2d 1385, 1395 

(2007); Sakraida v. AG Pro, Inc., 425 U.S. 273, 282, 
189 USPQ 449, 453 (1976); Anderson 's-Black Rock, 
Inc. v. Pavement Salvage Co., 396 U.S. 57, 62-63, 163 
USPQ 673, 675 (1969); Great Atlantic & P. Tea Co. v. 
Supermarket Equipment Corp., 340 U.S. 147, 152, 87 
USPQ 303, 306(1950). 



I. < OBVIOUSNESS REQUIRES ONLY A 
REASONABLE EXPECTATION OF SUC- 
CESS 

The prior art can be modified or combined to reject 
claims as prima facie obvious as long as there is a rea- 
sonable expectation of success. In re Merck & Co., 
Inc., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986) 
(Claims directed to a method of treating depression 
with amitriptyline (or nontoxic salts thereof) were 
rejected as prima facie obvious over prior art disclo- 
sures that amitriptyline is a compound known to pos- 
sess psychotropic properties and that imipramine is a 
structurally similar psychotropic compound known to 
possess antidepressive properties, in view of prior art 
suggesting the aforementioned compounds would be 
expected to have similar activity because the struc- 
tural difference between the compounds involves a 
known bioisosteric replacement and because a 
research paper comparing the pharmacological prop- 
erties of these two compounds suggested clinical test- 
ing of amitriptyline as an antidepressant. The court 
sustained the rejection, finding that the teachings of 
the prior art provide a sufficient basis for a 
reasonable expectation of success.); Ex parte Blanc, 
13 USPQ2d 1383 (Bd. Pat. App. & Inter. 1989) 
(Claims were directed to a process of sterilizing a 
polyolefinic composition with high-energy radiation 
in the presence of a phenolic polyester antioxidant to 
inhibit discoloration or degradation of the polyolefin. 
Appellant argued that it is unpredictable whether a 
particular antioxidant will solve the problem of dis- 
coloration or degradation. However, the Board found 
that because the prior art taught that appellant's pre- 
ferred antioxidant is very efficient and provides better 
results compared with other prior art antioxidants, 
there would have been a reasonable expectation of 
success.). 
> 

II. < AT LEAST SOME DEGREE OF PRE- 
DICTABILITY IS REQUIRED; APPLI- 
CANTS MAY PRESENT EVIDENCE 
SHOWING THERE WAS NO REASON- 
ABLE EXPECTATION OF SUCCESS 

Obviousness does not require absolute predictabil- 
ity, however, at least some degree of predictability is 
required. Evidence showing there was no reasonable 
expectation of success may support a conclusion of 
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DETAILED ACTION 

Claim Rejections - 35 USC § 103 

1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. Claims 24-32, 39-40, and 42-50 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Everett, Jr. et al. (US 5,202,661) in view of Hyuga (US 5,818,733). 

Re claims 24, 27, and 31, Everett teaches a system (figures 1, 2, 4, and 6) comprising: 
a communicator (16 and 20 of fig. 1, see also 16 and 20c of Jig. 2) configured to receive 
first data associated with an object (note the fixed sensor system, 12 of fig. I, detects the presence 
of an intruder that is considered as the first data associated with an object; col. 6, tines 24-40; 
col. 22, lines 35-50; col. 28, lines 50-64) and second data associated with the object (note the 
mobile sensor, 19 of fig. 2, detects the presence of the object that is considered as second data 
associated with the object; col. 13, lines 5-14; col. 14, lines 1-14, 20-29; 14, line 55-col. 15, line 
7), wherein the first data is received from a fixed detector configured to detect the first data (12 
of fig. 2, the fixed sensor system), and wherein the second data is received from a mobile target 
unit (18 of fig. 1, the mobile robot, fig. 6) comprising a sensor (19 of jig. 1, the mobile sensor 
system, see figure 4 for more details) configured to detect the second data; and 
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a processor (14 of figs. 1 and 2) configured to correlate the first data and the second data 
to generate object location information (col. 2, lines 44-64, col. 3, lines 35-42, 49-56; col. 24, 
lines 39-col. 25, line 4; col. 28, lines 50-65, note Cross correlation between the fixed sensor 
system, 1.2 of fig. 2, and the mobile sensor system, 19 of figs. I and 4, to determine an intruder at 
position (X, Y) as object location information; wherein the (X,Y) position of the intruder depicted 
in a floor plan map, col. 29, lines 5-9). 

It is noted that Everett teaches that the mobile target unit has the propulsion module (416 
of fig. 6) for moving the mobile target unit, and the propulsion module (426 of fig. 6) carried the 
camera and other elements as shown in figure 6, so this would obviously suggest that the mobile 
target unit includes a vehicle (416 and 422 of fig. 6) to move the mobile target unit. 

However, Everett does not particularly disclose wherein the mobile target unit is at least 
one of: mounted in the object, mounted on the object, carried in the object, or carried on the 
object, and the object is a vehicle as claimed. 

Hyuga teaches the mobile target unit (1 of fig. 2) is at least one of: mounted in the object 
(the mobile unit is carried by said sender or player or user), mounted on the object, carried in the 
object, or carried on the object (1 of fig. 2, the mobile unit can be held by the golf player, fig. 4, 
and carried on the golf cart, 29 of fig. 2) and the object is a vehicle (the golf cart, 29 of fig. 2). 

Therefore, taking the teachings of Everett and Hyuga as whole, it would have been 
obvious to one of ordinary skill in the art to modify the teachings of Hyuga into the system of 
Everett for the same purpose of generating the accuracy object location to improve monitoring 
object. 
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Re claim 25, Everett further teaches wherein the mobile target unit (18 of fig. 1, the 
mobile robot) comprises a locator unit (402 of fig. 4, note the local processor, 402 of fig. 4, 
passes required position and sonar information to the host computer, col. 7, lines 63 -col 8, lines 
25) configured to determine a target unit location (Note the local processor receives X- Y position 
and heading from processor, 417 of fig. 4, of propulsion module, 416 of fig. 4, which is 
considered as a determined a target location), the communicator (16, 20a, and 20c of fig. 2) 
being further configured to receive the target unit location (note the local processor, 402 of fig. 
4, passes required positional and sonar information to the host computer, 14 of fig. 1), the 
processor (14 of fig. 1) being further configured to determine whether the mobile target unit is 
within a range of the fixed detector (fig. 14, A-B indicates the mobile robot will be travel, so the 
mobile robot is within the range of the fixed sensor system; col. 3, lines 9-24). 

Re claim 26, Everett further teaches wherein the object location information comprises at 
least one of object trajectory information (col. 2, lines 34-39, the intruder 's report is considered 
as object trajectory information) or object speed information (note the robot's mean forward 
velocity is adjusted as a function of range to the intruder, which means the mobile robot enables 
to determine the object speed information so that the mobile robot to follow the intruder, col. 29, 
lines 15-23); and the fixed detector provides an image of the object (12gl of fig. 2, the video 
camera captures an image of the intruder). 

Re claim 28, Everett further teaches a database (Data stored in the history file is 
considered as database, col. 23, lines 40-55, col. 23, line 65-col 24, line 18) configured to 
maintain a plurality of current positions associated with at least one of a plurality of sensors, a 
plurality of mobile target units, or a plurality of objects. 
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Re claim 29, Everett further teaches wherein the mobile target unit comprises an 
accelerometer (41 7 of fig. 4, note velocity control and acceleration/deceleration ramping are 
performed by processor, 41 7) configured to provide data indicative of movement of the object to 
facilitate generating the object location information. 

Re claim 30, Everett further teaches wherein: the object is an identified good (Note the 
area under surveillance by the fixed and mobile sensors is considered as an identified good, col. 
21, line 64-col. 22, line 2); the mobile target unit (18 of fig. 2, the mobile robot) comprises a 
radio-frequency identification device (20b 1 and 20b2 of fig, 2); and the fixed detector (12 of fig. 
2) comprises a camera (I2gl of jig. 2) for observing the identified good (the area under 
surveillance), to facilitate thereby enabling the sensor (19 of fig. 4) and the fixed detector (12 of 
fig. 2) to provide corroborative surveillance of the identified good (col. 22, lines 35-50, see also 
col. 29, lines 1-9). 

Re claim 39, Everett further teaches wherein the mobile target unit (18 of fig. !, the 
mobile robot) comprises a locator unit (402 of fig. 4) coupled to determine a target unit location 
(col. 8, lines 8-25), the second data comprising the target unit location (col. 8, lines 8-25). 

Re claim 40, Everett further teaches wherein the correlating the first data and the second 
data comprises determining compliance with a scheduled object activity (function of time, col. 
13, lines 1-13). 

Re claim 42, Everett further teaches a plurality of detectors (12gl of figs. 2 and 4) each 
having a corresponding observation range (12gl of fig. I, note the video camera, 12g! of fig. 2, 
has a corresponding to observation range), wherein at least one of the plurality of detectors is 
selected to observe the object (12gl of fig. 4, the video camera , 12gl, follows the intruder), the 
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fixed detector (12gl of fig. 2, col. 29, lines 1-9) being selected in response to the processor's 
correlation of the first data and the second data (col. 28, lines 58-64). 

Re claim 43, Everett further teaches wherein the first data comprises at least one of an 
image of the object (I2gl of fig, 2, the video camera captures image of the intruder) or an 
identifier associated with the object. 

Re claim 44, Everett further teaches wherein the first data comprises a plurality of images 
of the object captured at different times (Note video signal from the video camera, 12gl of fig. 2, 
have images of the object at different, times). 

Re claim 45, Everett further teaches wherein the second data comprises at least one of an 
image of the object (e.g. 19h of fig. 6, the video camera captures image of the intruder) or an 
identifier associated with the object. 

Re claim 46, Everett further teaches wherein the second data comprises a plurality of 
images of the object captured at different times (note the video camera.. !9h of fig. 6, inherently 
captures images of the intruder at different times). 

Re claim 47, Everett further teaches wherein the object location information is 
determined at least in part based on a fixed detector location (e.g. 12 of fig. 2). 

Re claim 48, Everett further teaches wherein the object location information is 
determined at least in part based on a mobile target unit location (18 of fig. 2). 

Re claims 32 and 49 , Everett further teaches a movement module (col. 28, line 55-57, 
note the threat level is sufficient for the sofbvarefthe software is performed by the computer is 
considered as a movement module) to activate secondary sensors, and the ultrasonic motion 
detection system, 19fof fig. 4, is enabled) configured to activate a second fixed detector (note the 
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secondary sensors are activated) in response to the object location information (Note thread 
level includes the object location information), wherein the fixed detector (e.g. 12e of fig. 2) is 
further from the second fixed detector (e.g. 19e of fig. 4) than from a third fixed detector (12d of 
fig. 2). 

3. Claims 24-33 and 39-53 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Moengen (US 6,373,508). 

Re claims 24 and 31, Moengen teaches a system (figs. 1 and 2) comprising: 
a communicator (M, BL, and Q of fig. 2) configured to receive first data (e.g. Dl and Kl 
of fig. I, note transponders are also preferably provided both in the position detectors and the 
cameras, col. 12, lines 25-31) associated with an object (a natural object N, col. 3, lines 37-42) 
and second data associated with the object (mobile cameras are used, col. 12, lines 27-31; and 
col. 15, lines 10-38), wherein the first data is received from a fixed detector (Dl and Kl of fig. 1: 
see Dl of fig. 2) configured to detect the first data (note the position and image of the natural 
object is considered as the first data, e.g. Jigs. 3a-d) , and wherein the second data is received 
from a mobile target unit (note the mobile camera is used; col. 12, lines 27-31; and col. 16, lines 
10-38) comprising a sensor (a GPS is equipped with the mobile camera, col. 16, lines 11-20) 
configured to detect the second data (the position of natural object is determined by other means, 
col. 16, lines 12-20, the GSP is wirelessly connected to the position module M in figure 2); and, 

a processor (Q and P of fig. 1) configured to correlate (1 and 2 of fig. 2) the first data (the 
position of the natural object N is detected by the detector and camera, Dl and Kl of fig. 1) and 
the second data (the position of the natural object N is detected by the GPS, which is in afield of 
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view of a camera as the mobile unit, col. 16, lines 11-20) to generate object location information 
(e.g. Jigs. 9a, b, c, note the system for manipulating (4 of fig. 2) the picture of at least one 
movable, natural object in a natural television picture in such a manner that the object's position 
and movement are clearly visible in the television picture, col. 4, lines 51-63; wherein synthetic 
object represent position of the natural object any l time, which indicates the future position of 
the natural object). 

Moengen further teaches the object is a vehicle (a natural object would obviously 
considered a vehicle such as car race); and the mobile target unit (e.g. mobile camera include 
GPS. col. 16, lines 11-20, the GPS is equipped with the natural object) at least on of: mounted in 
the object, mounted on the object, carried in the object, or carried on the object (col. 10, lines 10- 
14). 

Re claims 25 and 39, Moengen further teaches wherein the mobile target unit (note the 
mobile camera, col. 16, lines 18-20) comprises the communicator (Q of figs. 1 and 2) is further 
configured to receive the target unit location, the processor (Q and P of fig. 1, the processing 
units 1-2 of jig. 2) being further configured to determine whether the mobile target unit (the 
mobile camera can be determined by the GPS) is within a range of the fixed detector (the mobile 
camera with the GPS within the field of view of a camera, e.g. Kl, K2, or K3 of fig. 1; col. 1 6, 
lines 12-14). 

Re claim 26, Moengen further teaches the object location information comprises at least 
one of object trajectory information (fig. 9c, S(p0), S(pl), S(p2) and S(p3)), or object speed 
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information (col. 9, lines 3-8); and the fixed detector provides an image of the object (e.g. Kl of 
fig. 1). 

Re claim 27, Moengen further teaches the object is a vehicle (a natural object would 
obviously be considered as a vehicle used in the car race event) (e.g. mobile camera include GPS, 
col. 16, lines 11-20, the GPS is equipped with the natural object). 

Re claim 28, Moengen further comprising a database (Note pre-stored code sequences as 
position and image of the natural object is considered as database, col. 11, lines 30-35) 
configured to maintain a plurality of current positions associated with at least one of a plurality 
of sensors (e.g Dl and D2 of fig. 1), a plurality of mobile target units (e.g. mobile cameras, col. 
12, lines 28-31), or a plurality of objects (natural objects: Nl and N2). 

Re claim 29, Moengen further teaches wherein the mobile target unit comprises an 
accelerometer (T of Jig. 2, note the active transponder also has to be mounted in the natural 
object, it has to be robust and capable of withstanding jolts and shocks as well as relative high 
accelerations) configured to provide data indicative of movement of the object to facilitate 
generating the object location information (col. 10, lines 3.0-14). 

Re claim 30, Moengen further teaches the object is an identified good (Note the natural 
object N is broadly interpreted as an identified good such as foot ball, hand ball, tennis ball, golf 
ball, and ice hockey pucks); the mobile target unit (the mobile camera, col. 15, lines 65-col. 16, 
lines 4) comprises a radio-frequency identification device (e.g. a wireless connection is 
inherently as a radio-frequency identification device, col. 16, lines 3-20); and the fixed detector 
(Dl of fig. 1) comprises a camera (Kl of fig. 1) for observing the identified good (e.g. the natural 
object N is within the field of view of the camera Kl of fig. 1), to facilitate thereby enabling the 
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sensor (e.g. GPS sensor is equipped into the mobile camera, col. 16, lines 12-20) and the fixed 
detector (note the mobile camera wirelessly communicates with the processing unit Q via data 
buses B, col. 16, lines 1-3) to provide corroborative surveillance of the identified good (e.g. Q 
and P of fig. 1, see figs. 9a-c). 

Re claim 32, Moengen teaches activating a second fixed detector (D2 and K2 of fig. 1) in 
response to the object location information (e.g. TK and D2 of fig. 1 , processed by the 
processing unit, 1 and 2 of fig. 2). 

Re claim 33, Moengen further teaches wherein the second data comprises an object 
identifier (Note the synthetic object S can be represented with various attributes for size, shape 
and color, which is considered as an object identifier, col. 7, lines 37-47), the method further 
comprising registering the object identifier (e.g. various attributes for size, shape and color of 
synthetic object S is pre-stored) in a database (e.g. Q and P of fig. 1) to indicate association with 
the object (figs. 9a-c). 

Re claim 40, Moengen further teaches wherein the correlating the first data and the 
second data comprises determining compliance with a scheduled object activity (e.g. figs. 3a-4d, 
x, y, z, t). 

Re claim 41, Moengen further teaches wherein the correlating the first data and the 
second data comprises determining a movement vector (col. 4, lines 26-49) to predict a future 
location of the object (not the speed, direction and the position of the natural object is shown on 
display, e.g. P of fig. 1, indicating the future location of the object). 

Re claim 42, Moengen further teaches further comprising a plurality of detectors (TK, 
Dl, Kl; TK, D2, K2; and TK, D3, K3 of fig. 1, plurality cameras) each having a corresponding 
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observation range, wherein at least one of the plurality of detectors is selected to observe the 
object (note a preselected x,y,z co-ordinate system at the time t, which means one of the camera 
is selected, K2 of fig. 1, see fig. 3b), the fixed detector being selected in response to the 
processor's correlation of the first data and the second data (e.g. 3 of fig. 2, camera control 
system, based on the correlating performed by the processor, 0, 1, 2, and 4 of fig. 2). 

Re claim 43, Moengen further teaches wherein the first data comprises at least one of an 
image of the object (e.g. 1K2 of fig. 3 A) or an identifier associated with the object (Dl of fig. 1). 

Re claim 44, Moengen further teaches wherein the first data comprises a plurality of 
images of the object captured at different times (IK2 of fig. 3A and I'K2 of fig. 3c). 

Re claim 45, Moengen further teaches wherein the second data (e.g. K3 of fig. 1 as the 
mobile camera) comprises at least one of an image of the object (e.g. IK3 of fig. 4a) or an 
identifier associated with the object (D3 of fig.l, and GPS is equipped with the mobile camera). 

Re claim 46, Moengen further teaches wherein the second data comprises a plurality of 
images of the object captured at different times (figs. 4a and 4c). 

Re claim 47, Moengen further teaches wherein the object location information is 
determined at least in part based on a fixed detector location (VZ2, and VZ2 of figs. 3a and 3c). 

Re claim 48, Moengen further teaches wherein the object location information is 
determined at least in part based on a mobile target unit location (GPS system determines the 
location of the natural object). 

Re claim 49, Moengen further teaches a movement module configured to activate a 
second fixed detector (e.g. TK, D2, K2 of fig. 1) in response to the object location information (1 
and 2 of fig. 2). 
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Re claim 50, Moengen further teaches wherein the correlating the first data and the 
second data to generate the object location information comprises determining at least one of a 
trajectory or a speed of the object (col. 9, lines 3-7; col. 14, lines 33-36). 

Re claim 5 1 , Moengen further teaches wherein the mobile target unit comprises a locator 
unit configured to determine the target unit location mobile (a position of the mobile camera is 
determined by the GPS; col. 16, lines 11-17). 

Re claim 52, Moengen further teaches wherein the fixed detector is configured to be 
selected in response to the processor's correlation of the first data and the second data processor's 
correlation of the first data and the second data (e.g. 3 of fig. 2, camera control system based on 
the correlating performed by the processor Q, 1, 2, and 4 of fig. 2). 

Re claim 53, Moengen further teaches wherein the fixed detector is further from the 
second fixed detector than from a third fixed detector (TK, D3, K3 of fig. 1). 

4. Claim 41 is rejected under 35 U.S.C. 103(a) as being unpatentable over Everett, Jr. et al. 
(US 5,202,661) in view of Hyuga (US 5,818,733) and further in view of Kitamura et al. 
(5,554,983). 

Re claim 41, Everett and Hyuga does not particularly disclose wherein the correlating the 
first data and the second data comprise determining a movement vector to predict a future 
location of the object as claimed. 

Kitamura teaches wherein the correlating the first data and the second data (fig. 6) 
comprise determining a movement vector to predict a future location of the object (fig. 8). 
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Taking the teachings of Everett, Hyuga, and Kitamura as a whole, it would have been 
obvious to one to modify the teachings of Kitamura into the combined of Everett and Hyuga to 
reduce the processing time and accurately predict the object's position. 

Response to Arguments 

5 . Applicant's arguments filed 07/36/2009 have been fully considered but they are not 
persuasive. 

The applicant argued that Everett teaches against " wherein the mobile target unit is at 
least one of: mounted in the object, mounted on the object, carried in the object, or carried on the 
object" as amended in claim 1 . 

The examiner respectfully disagrees with the applicant. It is submitted that Everett 
teaches that the mobile target unit (18 of fig. 6) has the propulsion module (416 of fig. 6) for 
moving the mobile target unit (1 8 of fig. 6), and the propulsion module (426 of fig. 6) carried the 
camera and other elements as shown in figure 6, so this would obviously suggest that the mobile 
target unit includes a vehicle (416 and 422 of fig. 6) for the mobile target unit is mounted on in 
order to move the mobile unit. 

Hyuga teaches the mobile target unit (1 of fig. 2) is at least one of: mounted in the object 
(a mobile unit carried by said sender or player or user), mounted on the object, carried in the 
object, or carried on the object (1 of fig. 2, the mobile unit can be hold by the golf player, fig. 4, 
and carried on the golf cart, 29 of fig. 2) and the object is a vehicle (the golf cart, 29 of fig. 2). 
With the suggested teachings of Everett and Hyuga, one skilled in the art would combine 
Everrett and Hyuga to make obvious claimed features to improve monitoring object. 
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' The applicant further argues that Moengen does not disclose "a processor configured to 
correlate the first data and the second data to generate object location information" as claimed. 

The examiner respectfully disagrees with the applicant. It is submitted that Moegen 
teaches a processor (Q and P of fig. 1) configured to correlate (1 and 2 of fig. 2) the first data 
(the position of the natural object N is detected by the de tector and camera, Dl and K.1 of fig. I ) 
and the second data (the position of the natural object N is detected by the GPS, which is in a 
field of view of a camera as the mobile unit, col. 16, lines 1 1-20) to generate object location 
information (e.g. figs. 9a, b, c, note the system for manipulating (4 of fig. 2) the picture of at 
least one movable, natural object in a natural television picture in such a manner that the object's 
position and movement are clearly visible in the television picture, col. 4, lines 51-63; wherein 
synthetic object represent position of the natural object any t time, which indicates the future 
position of the natural object). Note claim features do not include any image that is used to 
generate object location. Therefore, the disclosure of Moengen meets the claimed features. 

Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 . 1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
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CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Contact Information 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Tung Vo whose telephone number is 571-272-7340. The 
examiner can normally be reached on Monday- Wednesday, Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mehrdad Dastouri can be reached on 571-272-741 8. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

/Tung Vo/ 

Primary Examiner, Art Unit 2621 
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IN SUPPORT OF APPELLANTS' APPEAL 
TO THE BOARD OF PATENT APPEALS AND INTERFERENCES 



In reply to the Notification of Non-compliant Appeal Brief, mailed November 8, 2006, 
Appellants hereby submit the following Amended Appeal Brief pursuant to 37 CFR 41 .37 in 
support of an appeal from the final rejection by the Examiner, dated May 24, 2006 and advisory 
action dated June 23, 2006, in the above-captioned case. Appellants filed a notice of appeal 
under § 41 .31 on August 24, 2006. Also, Appellants assert that the following brief does not 
include any new or non-admitted amendment, affidavit, or other evidence. Appellants 
respectfully request consideration of this appeal by the Board of Patent Appeals and 
Interferences for allowance of the above-captioned patent application. 
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I. Real party in interest 

The real party in interest is Dennis Fernandez, an individual, having a residence at 
1 175 Osborn Avenue, Atherton, CA 94027. 

II. Related appeals and interferences 

Related proceedings to this application include US Patent Application No. 
09/823,089 with Appeal Brief filed on June 12, 2006 and Application No. 09/823,508 
with Notice of Appeal filed on October 1 1 , 2006. To the best of Appellant's knowledge, 
there are no other prior or pending appeals, judicial proceedings or interferences known 
to the appellant which may be related to, directly affect or be directly affected by or have 
a bearing on the Board's decision in the pending appeal. 

III. Status of Claims 

In Examiner's Advisory Action Before the Filing of an Appeal Brief, mailed June 
23, 2006, all claims 1-23 were rejected. Claims 1-23 are currently pending in this 
proceeding, Claims 1,11 and 20 being independent claims. These pending claims, 1-23, 
are being appealed, and are appended herewith in the Claims appendix. 

IV. Status of Amendments 

In the Final Office Action mailed May 24, 2006, Examiner rejected claims 1-23. 
Subsequent to this final rejection, Applicant responded with an amendment filed June 2, 
2006. Examiner entered the proposed amendments, in the Advisory Action mailed June 
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23, 2006, while affirming the rejection of claims 1-23. All pending claims 1-23 on 
appeal are provided in the Claims appendix, as filed in the June 2 amendment. 

V. Summary of claimed subject matter 

The subject matter of the invention, defined in independent claims 1, 1 1 and 20, is 
related to remote surveillance and communications technology, particularly to integrated 
fixed and mobile network electronics and related software for object attribute processing 
(see Appellant's Specification, page 1, lines 7-9; page 2, lines 19-20; and FIG. 1). 

Specifically, the claimed subject matter pertains to goods inventory management 
(page 27, lines 19-27; FIG. 3, item 167 "Report") via the internet (page 9, lines 9-18) and 
through utilizing fixed detectors (page 6, lines 7-14; page 7, line 1-page 8, line 20; FIG. 
1, item 3 "Detector;" page 34, lines 24-29; and page 35, lines 4-8) and mobile sensors 
(page 6, lines 7-14; page 34, lines 24-26; and page 35, lines 1-8). Monitored objects 
(page 3, lines 1-7 and 12-19; and FIG.l, item 2 "Object") are represented with a data 
structure (page 2, lines 1 1-13; and page 18, lines 25-30) comprising of 1) an object 
identifier (page 20, lines 7-17) representing one or more goods in production, inventory 
and shipment; 2) a first object location and time (page 22, lines 7-13), provided by a 
detector coupled to the console processing unit; and 3) a second object location and time, 
provided by a sensor coupled to the console processing unit. Additionally, the invention 
discloses a method (page 2, lines 28-29; FIG. 4; page 9, lines 9-18; page 29, lines 17-24; 
and page 30, lines 4-13) for processing such data structure and the utilization of a single- 
chip integrated circuit for doing so. 
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Security is addressed in the invention as an access means processes the data 
structure securely using a digital certificate (page 22, lines 15-23), watermark (page 8, 
lines 14-16) or encryption key (page 22, lines 15-17). This security feature assures that 
such data structure is accessible, for object-monitoring from only one or more specified 
network site or processor. Control software is implemented in order to provide the data 
structure (page 2, lines 11-13) automatically and for network surveillance in response to 
user search query (page 1 1 , lines 9-16; page 14, line 29-page 1 5, line 1 5; and page 31, 
line 12-18). Furthermore, such software is comprised of numerous components including 
a network and data communication module (page 14, line 22; and FIG. 3, item 161 
"Communication"), an object and map database (page 14, lines 6-7; FIG. 3, item 162 
"Database"; and page 18, lines 17-23) an object movement processing module (page 14, 
line 7, FIG. 3, item 163 "Movement"; and page 19, lines 12-29), a security management 
module (page 14, line 7; FIG. 3,. item 164 "Security"; and page 22, line 15-page 23, line 
8), an electronic transaction processing module (page 14, line 8; FIG. 3, item 165 
"Transaction"; and page 23, line 25-page 25, line 1 1), a diagnosis tool (page 14, line 8; 
FIG. 3, item 166 "Diagnosis"; and page 26, lines 15-22), and a performance report 
updater module (page 14, lines 8-9; FIG. 3, item 167 "Report"; and page 27, line 19-page 
28, line 2). Also included is a visual object analyzer module (page 14, line 9; FIG. 3, 
item 168 "Visual"; and page 28, lines 4-15) comprising a neural network or simulation 
program for recognizing adaptively one or more identified goods for real-time tracking of 
multiple goods movement, whereby such modules are functionally integrated to enable 
surveillance-based commercial transaction using the data structure (page 25, lines 13-28; 
and page 27, lines 1-7). 
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VI. Grounds of rejection to be reviewed on appeal 

A. Whether claims 1-3, 5-12, 14-16, 18-21, and 23 are unpatentable under 35 
U.S.C. 103(a) over Rauber et al. (US 6,182,053) in view of Woolston (US 
5,845,265). 

B. Whether claims 13, 17, and 22 are unpatentable under 35 U.S.C. 103(a) over 
Rauber et al. (US 6,182,053) in view of Woolston (US 5,845,265) as applied 
to claims 1,11, and 20, and further in view of Durbin et al. (US 6,039,258). 

C. Whether claim 4 is unpatentable under 35 U.S.C. 103(a) over Rauber et al. 
(US 6,182,053) in view of Woolston (US 5,845,265) as applied to claim 1, 
and in further view of Kennedy (US 6,301,480). 

VII. Argument 

A. The Examiner has clearly failed to provide a basis or rationale for the 
rejection of Claim 3, therefore Examiner has not established a prima facie 
case of obviousness under 35 U.S.C. 103(a). 

In order to establish a prima facie case of obviousness, the prior art references 
when combined must teach or suggest all the claim limitations. See, MPEP 706.020). 
Examiner begins the rejection of Appellant's invention by concluding that claims 1-3, 5- 
12, 14-16, 18-21, and 23 are unpatentable under 35 U.S.C. 103(a) over Rauber et al. (US 
6,182,053) in view of Woolston (US 5,845,265). Thereafter, Examiner states Appellant's 
claim language in each of said claims with reference to specific figures and text of 
Rauber and Woolston. However, Examiner has failed to indicate which prior art 
reference and more specifically, where in that reference the language of Claim 3 is 



Application No. 09/823,509 



6 ATTORNEY DOCKET: FERN-P00 1 C 

Appellant's Brief 

Related Proceedings Appendix, Page 172 



taught. Nowhere in Examiner's argument is there provided a reference to prior art which 
teaches, "a position signal being generated by the detector coupled to the monitored 
object when such object is moveable within an observable range, a visual signal being 
generated by the sensor uncoupled to such object in the observable range," (Appellant's 
Claim 3). 

Where a reference is relied on to support a rejection, whether or not in a minor 
capacity, that reference should be positively included in the statement of the rejection. In 
re Hoch, 428 F.2d 1341, 1342 (C.C.P.A. 1970). Additionally, MPEP 706.020) sets forth 
the contents of an Examiner's 35 U.S.C. 103 rejection as including, "the relevant 
teachings of the prior art relied upon, preferably with reference to the relevant column or 
page number(s) and line number(s) where appropriate." In addressing Appellant's 
specific claim language that pertains to Rauber and similarly in addressing Appellant's 
specific claim language that pertains to Woolston, Examiner has failed to indicate the 
proper citation as to where the alleged § 103(a) rejection of Claim 3 is based. MPEP 
706.02(j) states that, "[i]t is important for an examiner to properly communicate the basis 
for a rejection so that the issues can be identified early and the applicant can be given a 
fair opportunity to reply." 

Appellant argues that Examiner has provided no basis for rejection under § 103(a) 
due to the failure to cite in the prior art references the alleged teaching of Claim 3's 
language. Examiner has merely asserted rejection of Claim 3. Therefore, without 
rationale for the § 103(a) rejection of Claim 3, Examiner has not proven that the prior art 
references teach or suggest all the claim limitations under MPEP 706.020), and thus has 
failed to establish a prima facie case of obviousness. 
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B. The Examiner has not provided sufficient evidence of prima facie 

obviousness under §103(a) for the rejection of Claims 1-3, 5-12, 14-16, 18- 
21, and 23 as unpatentable over the prior art of Rauber in combination 
with Woolston. 

On appeal to the Board, an applicant can overcome a rejection by showing 
insufficient evidence of prima facie obviousness. In re Roujfett, 149 F.3d 1350, 1355 
(Fed. Cir. 1998). Examiner has argued that Claims 1-3, 5-12, 14-16, 18-21, and 23 are a 
combination of elements found in Rauber and Woolston. It is known that most, if not all, 
inventions are combinations and mostly of old elements. Stratoflex, Inc. v. Aeroquip 
Corp., 713 F.2d 1530, 1540 (Fed. Cir. 1983). In demonstrating the combination, 
Examiner has merely identified relevant text and figures in both Rauber and Woolston 
that correspond to Appellant's Claim language. However, the court has held that mere 
identification of each element in the prior art is insufficient to defeat the patentability of 
the combined subject matter as a whole. Rouffett, 149 F.3d at 1355, 1357. To establish a 
prima facie case of obviousness based on a combination of elements disclosed in the 
prior art, the Examiner must articulate the basis on which it concludes that it would have 
been obvious to make the claimed invention. Id. Specifically, Examiner must "explain 
the reasons why one of ordinary skill in the art would have been motivated to select the 
references and to combine them to render the claimed invention obvious." Id. at 1359. 
In detail, this "motivation-suggestion-teaching" test asks not merely what the references 
disclose, but whether a person of ordinary skill in the art, possessed with the 
understandings and knowledge reflected in the prior art, and motivated by the general 
problem facing the inventor, would have been led to make the combination recited in the 
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claims. See, Cross Med. Prods., Inc. v. Medtronic Sofamor Danek, Inc., 424 F.3d 1293, 
1321-24 (Fed. Cir. 2005). 

In order to find evidence of such motivation, suggestion or teaching, the court has 
recommended three places: 1) the content of the public prior art; 2) the nature of the 
problem addressed by the invention; or 3) the knowledge of one of ordinary skill in the 
art. See, SIBA Neurosciences, Inc. v. CadusPharm. Corp., 225 F.3d 1349, 1356 (Fed. 
Cir. 2000). Examiner has presumably addressed the second place, the nature of the 
problem addressed by the invention. In determining whether the nature of the problem 
addressed by the invention supplies a motivation to combine certain prior art references, 
the court has engaged in a detailed analysis of the nature of the problem solved by the 
invention. See, Princeton Biochemicals, Inc. v. Beckman Coulter, Inc., 41 1 F.3d 1332, 
1338-39 (Fed. Cir. 2005). Here the Examiner has not even began to provided detailed 
analysis of the sort of problem addressed by Appellant's invention. Instead, Examiner 
briefly states that the alleged motivation to combine Rauber and Woolston by one of 
ordinary skill in the art would allow for, "observing goods with its price during 
inventory," and "allow the user to easily set up his or her own warehouse, store, or 
retailer for buying and selling goods via the Internet." In other words, Examiner is 
arguing that the nature problem addressed by Appellant's invention is the setting up of 
warehouses for internet sales and observing goods with price during inventory. 
Examiner argues the combination of Rauber and Woolston is motivated because it allows 
the user to "easily set up" such internet retail operations. Appellant argues that such 
reasoning is flawed.- 
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First, Examiner misses the point of the nature of the problem addressed by 
Appellant's invention. In short, Appellant's invention deals with the problem of goods 
inventory management via the internet and through utilizing fixed detectors and mobile 
sensors, not the problem as stated by Examiner. Furthermore, if a person or ordinary skill 
in the art was faced with the problem as articulated by Examiner of setting up warehouses 
for internet sales with observation of goods with price during inventory, and that such 
person wanted to solve this problem "easily" as examiner states, they would not need to 
combine Rauber and Woolston. One would merely need to utilize the prior art as taught 
by the Woolston patent. In this patent, Woolston teaches a method for creating a 
"computerized market for used and collectible goods," i.e., Internet sales. Also, 
Woolston teaches a method to allow the "purchaser to change the price of the good once 
the purchases has purchased the good thereby to allow the purchaser to speculate on the 
price." In fact, this patent allowed Woolston to found the company MercExchange, 
L.L.C in 2001 and successfully commercialize this on-line auction technology. The 
Woolston patent essentially solved the problem of setting up warehouses for internet 
sales with observation of goods with price during inventory. Appellant stresses that 
Woolston was able to do this solely in reliance on his own patented invention and without 
the combination of the Rauber patent. Therefore, a person of ordinary skill in the art 
faced with the problem of "easily" setting up warehouses for internet sales with 
observation of goods with price during inventory would only need to utilize the Woolston 
patent. 

Furthermore, Appellant argues that a person of ordinary skill in the art would not 
rely on the Rauber patent to solve such a problem. It has been long established by the 
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court that in order to be considered in a § 103(a) obviousness analysis, prior art references 
must be analogous art. See, Application ofAntle, 444 F.2d 1 168 (C.C.P.A. 1971). The 
characteristic of being analogous refers to the prior art that courts deem, as a matter of 
law, a person having ordinary skill in the art would reasonably have consulted in solving 
the problem addressed by the claimed in invention. Rauber is non-analogous art to the 
problem, as articulated by Examiner, of "easily" setting up warehouses for internet sales 
with observation of goods with price during inventory. Rauber relates to managing 
distressed inventory, which is defined as inventory that is shipped by freight carriers 
which does not reach its intended destination, is not accepted by the buyer, accidentally 
loaded on the wrong freight truck, damaged, or simply marked improperly. Thus, it is 
evident that Rauber does not relate to the nature of the problem as articulated by 
Examiner. Therefore, Rauber is non-analogous prior art which does not address the 
particular problem of "easily" setting up warehouses for internet sales with observation of 
goods with price during inventory. 

With only a need to utilize the Woolston patent to solve the nature of the problem 
as articulated by the Examiner, Appellant can find no teaching, suggestion or motivation 
to combine the prior art of Woolston with Rauber. Therefore, absent such findings, 
Appellant concludes that Examiner has stated insufficient evidence of prima facie 
obviousness of Claims 1-3, 5-12, 14-16, 18-21 and 23 under §103(a), and as such 
Appellant's application should be allowed. 
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C. The Examiner has made contradictory and incorrect statements and thus 
has improperly rejected claims 1-3, 5-12, 14-16, 18-21, and 23 under 35 
U.S.C. 103(a) as being unpatentable over Rauber and Woolston. 

The language of independent Claims 1, 1 1 and 20 state in part, "a first object 
location and time monitored at such location, the first object location being provided by a 
detector" (italics added). In addressing this language, Examiner cites Rauber (fig. 2, 
items 212-218) and states that the warehouse would obviously have a detector for 
detecting the first object. 

First, Appellant presumes that Examiner is arguing that the warehouse inherently 
has a detector. If that is the case, Examiner cannot merely assert such an inherency 
determination. Rather, the examiner must provide a basis in fact and/or technical 
reasoning to reasonably support the determination that the allegedly inherent 
characteristic necessarily flows from the teachings of the applied prior art." Ex parte 
Levy, 17 U.S.P.Q.2d 1461, 1464 (Bd. Pat. App. & Inter. 1990). Examiner has not done 
so here and thus has incorrectly stated that such warehouse in Rauber would, "obviously 
have a detector." 

Second, after stating that such detector is obvious in the warehouse of Rauber, 
Examiner then states, "[i]t is noted that Rauber does not particularly teach Internet and at 
least one fixed detector?' Here Examiner has done an about face - first in stating that 
Rauber would obviously have a detector, and then stating that Rauber does not have such 
detector. These are contradictory statements. 

Therefore, Appellant argues that Examiner has improperly rejected claims 1-3, 5- 
12, 14-16, 18-21, and 23 under 35 U.S.C. 103(a) as being unpatentable over Rauber and 
Woolston and as such Appellant's application should be allowed. 
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D. Without establishing a prima facie case for the §103(a) obviousness 

combination of Rauber and Wooiston, Examiner has not established that 
Claims 13, 17, and 22 are unpatentable over Rauber and Wooiston in 
view of Durbin et al., and that Claim 4 is unpatentable in view of 
Kennedy. 

In rejecting claims 13, 17 and 22, Examiner relies on the combination of Rauber 
and Wooiston with the prior art reference of Durbin et al. Appellant has previously 
argued, extensively, that there is no teaching, suggestion or motivation to combine the 
prior art of Wooiston with Rauber, and thus Examiner has stated insufficient evidence for 
the prima facie obviousness combination of these prior art references under §103(a). 
Therefore, without a valid combination of Rauber and Wooiston, Examiner's argument 
for rejecting Claims 13, 17 and 22 via incorporation of the teachings of Durbin into the 
combined system of Rauber and Wooiston must fail. Similarly, with respect to Claim 4, 
Examiner's argument for the incorporation of the teachings of Kennedy into the system 
of Rauber and Wooiston must also fail. In conclusion, claims 4, 13, 17 and 22 are 
patentable over the Rauber, Wooiston, Durbin, and Kenney, and as such Appellant's 
application should be allowed. 
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CONCLUSION 



For all of the reasons stated above, Appellant respectfully concludes that 
Examiner was in error to reject claims 1-3, 5-12, 14-16, 18-21, and 23 as being 
unpatentable over Rauber et al. (US 6,1 82,053) in view of Woolston (US 5,845,265) 
under 35 U.S.C. 103(a); Examiner was in error to reject claims 13,17, and 22 as being 
unpatentable over Rauber in view of Woolston as applied to claims 1,11, and 20, and in 
further view of Durbin et al. (US 6,039,265) under 35 U.S.C. 103(a); and Examiner was 
in error to reject claim 4 as being unpatentable over Rauber in view of Woolston as 
applied to claim 1, and in further view of Kennedy (US 6,301,480) under 35 U.S.C. 
103(a). Therefore, Appellant prays for careful consideration of this appeal by the Board 
of Patent Appeals and Interferences in order for the ultimate allowance of all Claims 1- 



23. 



Respectfully Submitted, 





JAMES HARRIS 
Reg. No. 52,995 



Fernandez & Associates, LLP 



Customer No. 22877 
Phone: (650) 325-4999 
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VIII. Claims appendix 



Claims Presented For Appeal 
(as filed via Rule-116 Amendment dated June 2, 2006) 

1. (PREVIOUSLY PRESENTED) In a console processing unit for goods inventory 
management coupled via the Internet to at least one fixed detector and at least one mobile 
sensor, a data structure for representing a monitored object, the data structure comprising: 
an object identifier, such object identifier representing one or more goods in 

production, inventory and shipment; 
a first object location and a time monitored at such location, provided by a 

detector coupled to the console processing unit; and 
a second object location and a time monitored at such location, provided by a 

sensor coupled to the console processing unit; 
wherein an access means processes the data structure securely using a digital certificate, 
watermark or encryption key, such that the data structure is accessible for object- 
monitoring from only one or more specified network site or processor, the data structure 
being provided automatically using control software for network surveillance in response 
to a user search query, the software comprising a network and data communication 
module, an object and map database, an object movement processing module, a security 
management module, an electronic transaction processing module, a diagnosis tool, a 
performance report updater module, and a visual object analyzer module comprising a 
neural network or simulation program for recognizing adaptively one or more identified 
goods for real-time tracking of multiple goods movement, whereby such modules are 
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functionally integrated to enable surveillance-based commercial transaction using the 
data structure. 

2. (PREVIOUSLY PRESENTED) The data structure of Claim 1 further comprising: 

a scheduled object location and a time scheduled for such location. 

3. (PREVIOUSLY PRESENTED) The data structure of Claim 1 wherein: 

a position signal being generated by the detector coupled to the monitored object 
when such object is moveable within an observable range, a visual signal being 
generated by the sensor uncoupled to such object in the observable range. 

4. (PREVIOUSLY PRESENTED) The data structure of Claim 1 wherein: 

the detector comprises an accelerometer. 

5. (PREVIOUSLY PRESENTED) The data structure of Claim 1 wherein: 

a software agent associated with the monitored object accesses a database. 

6. (PREVIOUSLY PRESENTED) The data structure of Claim 1 wherein: 

the object identifier comprises an object name, an object group, an object query, 
an object condition, an object status, an object location, an object time, an object 
error, or an object image, video, or audio broadcast signal. 
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7. (PREVIOUSLY PRESENTED) The data structure, of Claim 3 wherein: 

the observable range is modifiable according to a rule set. 

8. (PREVIOUSLY PRESENTED) The data structure of Claim 1 wherein: 

the monitored object is monitored temporarily using an extrapolated or last-stored 
positional or visual signal. 

9. (PREVIOUSLY PRESENTED) The data structure of Claim 1 wherein: 

the monitored object is authenticated according to a voice pattern, a finger-print 
pattern, a handwritten signature, or a magnetic or smart-card signal. 

10. (PREVIOUSLY PRESENTED) The data structure of Claim 1 wherein: 

the monitored object is provided an electronic. file comprising a book, a greeting 
card, a news report, a sports report, a stock report, an artwork, a research 
database, a personal list, a recorded or live voice or music transmission, an 
electronic tool, or a commercial transaction. 

1 1 . (PREVIOUSLY PRESENTED) In a console processing unit for goods inventory 
management coupled via the Internet to at least one fixed detector and at least one mobile 
sensor, a method for processing a data structure for representing a monitored object, the 
method comprising the step of: 

transmitting to a processor in a network a data structure comprising an object 
identifier, such object identifier representing one or more goods in production, 
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inventory and shipment, a first object location and a time monitored at such 
location, the first object location being provided by a detector coupled to a 
console processing unit, and a second object location and a time monitored at 
such location, the second object location being provided by a sensor coupled to 
the console processing unit; wherein an access means processes the data structure 
securely using a digital certificate, watermark or encryption key, such that the 
data structure is accessible for object-monitoring from only one or more specified 
network site or processor, the data structure being provided automatically using 
control software for network surveillance in response to a user search query, the 
software comprising a network and data communication module, an object and 
map database, an object movement processing module, a security management 
module, an electronic transaction processing module, a diagnosis tool, a 
performance report updater module, and a visual object analyzer module 
comprising a neural network or simulation program for recognizing adaptively 
one or more identified goods for real-time tracking of multiple goods movement, 
whereby such one or more of such modules are functionally integrated to enable 
surveillance-based commercial transaction using the data structure. 

12. (PREVIOUSLY PRESENTED) The data structure of Claim 1 wherein: 

the sensor comprises a radio-frequency identification device for locating the 
identified goods, and the detector comprises a camera for observing such 
identified goods, thereby enabling the sensor and the detector to provide 
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corroborative surveillance of the identified goods within an observable range in 
which the sensor is mobile relative to the detector. 

13. (PREVIOUSLY PRESENTED) The data structure of Claim 1 wherein: 

the sensor comprises a sensor signal port for sensing a low-power or fuel 
condition of the identified goods, thereby enabling the console processing unit to 
indicate or warn a down period for using the identified goods. 

14. (PREVIOUSLY PRESENTED) The data structure of Claim 1 wherein: 

the detector comprises visual-analyzer means for recognizing adaptively the 
identified goods using a neural network or simulation program, thereby enabling 
secure inventory management of the identified goods. 

15. (PREVIOUSLY PRESENTED) The data structure of Claim 1 wherein: 

the data structure indicates in-stock availability of the identified goods for 
transacting shipment, and a tax-rate for transaction at the location of the identified 
goods. 

16. (PREVIOUSLY PRESENTED) The method of Claim 11 wherein: 

the sensor comprises a radio-frequency identification device for locating the 
identified goods, and the detector comprises a camera for observing such 
identified goods, thereby enabling the sensor and the detector to provide 
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corroborative surveillance of the identified goods within an observable range in 
which the sensor is mobile relative to the detector. 

17. (PREVIOUSLY PRESENTED) The method of Claim 1 1 wherein: 

the sensor comprises a sensor signal port for sensing a low-power or fuel 
condition of the identified goods, thereby enabling the console processing unit to 
indicate or warn a down period for using the identified goods. 

18. (PREVIOUSLY PRESENTED) The method of Claim 11 wherein: 

the detector comprises visual-analyzer means for recognizing adaptively the 
identified goods using a neural network or simulation program, thereby enabling 
secure inventory management of the identified goods. 

19. (PREVIOUSLY PRESENTED) The method of Claim 1 1 wherein: 

the data structure indicates in-stock availability of the identified goods for 
transacting shipment, and a tax-rate for transaction at the location of the identified 
goods. 

20. (PREVIOUSLY PRESENTED) In a network for goods inventory management for 
coupling at least one fixed detector and at least one mobile sensor, a single-chip 
integrated circuit for processing a data structure for representing a monitored object, the 
circuit comprising: 

Application No. 09/823,509 20 ATTORNEY DOCKET: FERN-POO 1C 
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a processor provided in a wireless target unit for transmitting or receiving in a 
network a data structure comprising an object identifier, such object identifier 
representing one or more goods in production, inventory and shipment, a first 
object location and a time monitored at such location, the first object location 
being provided by a detector, and a second object location and a time monitored 
at such location, the second object location being provided by a sensor; wherein 
an access means processes the data structure securely using a digital certificate, 
watermark or encryption key, such that the data structure is accessible for object- 
monitoring from only one or more specified network site or processor, the data 
structure being provided automatically using control software for network 
surveillance in response to a user search query, the software comprising a network 
and data communication module, an object and map database, an object 
movement processing module, a security management module, an electronic 
transaction processing module, a diagnosis tool, a performance report updater 
module, and a visual object analyzer module comprising a neural network or 
simulation program for recognizing adaptively one or more identified goods for 
real-time tracking of multiple goods movement, whereby one or more of such 
modules are functionally integrated to enable surveillance-based commercial 
transaction using the data structure. 



21. (PREVIOUSLY PRESENTED) The circuit of Claim 20 wherein: 

the sensor comprises a radio-frequency identification device for locating the 
identified goods, and the detector comprises a camera for observing such 
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identified goods, thereby enabling the sensor and the detector to provide 
corroborative surveillance of the identified goods within an observable range in 
which the sensor is mobile relative to the detector. 

22. (PREVIOUSLY PRESENTED) The circuit of Claim 20 wherein: 

the sensor comprises a sensor signal port for sensing a low-power or fuel 
condition of the identified goods, thereby enabling indication or warning of a 
down period for using the identified goods. 

23. (PREVIOUSLY PRESENTED) The circuit of Claim 20 wherein: 

the detector comprises visual-analyzer means for recognizing adaptively the 
identified goods using a neural network or simulation program, thereby enabling 
secure inventory management of the identified goods. 
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VIII. Evidence appendix 

None. 

X. Related proceedings appendix 

None. 
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DETAILED ACTION 

Reopen Prosecution 

In view of the appeal brief filed on 1 1/17/2006, and the appeal conference with 
supervisors, Mehrdad Dastouri and Thai Tran, and the examiner, Tung Vo, on 02/27/2007, 
PROSECUTION IS HEREBY REOPENED. The office action is set forth below. 

To avoid abandonment of the application, appellant must exercise one of the following 
two options: 

(1) file a reply under 37 CFR 1 . 1 1 1 (if this Office action is non-final) or a reply under 37 
CFR 1.1 13 (if this Office action is final); or, 

(2) initiate a new appeal by filing a notice of appeal under 37 CFR 41.31 followed by an 
appeal brief under 37 CFR 41.37. The previously paid notice of appeal fee and appeal 

brief fee can be applied to the new. appeal. If, however, the appeal fees set forth in 37 
CFR 41 .20 have been increased since they were previously paid, then appellant must 
pay the difference between the increased fees and the amount previously paid. 

A Supervisory Patent Examiner (SPE) has approved of reopening prosecution by signing 

bel ° W c / //e<&i-<fbs/ 

MEHRDAD DASTOURI 
SUPERVISORY RATENT EXAMINER 
TC 

It is noted that that claims 1-10, and 12-15 are rejected under 35 U.S.C. 101, and claims 
1-23 are rejected as follows. 
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Claim Rejections - 35 USC §101 
1. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or 
any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 

Descriptive material can be characterized as either "functional descriptive material" or "nonfunctional descriptive 
material." In this context, "functional descriptive material" consists of data structures and computer programs 
which impart functionality when employed as a computer component. (The definition of "data structure" is "a 
physical or logical relationship among data elements, designed to support specific data manipulation functions." 
The New IEEE Standard Dictionary of Electrical and Electronics Terms 308 (5th ed. 1 993).) "Nonfunctional 
descriptive material" includes but is not limited to music, literary works and a compilation or mere arrangement 
of data. 

When functional descriptive material is recorded on some computer-readable medium it becomes structurally and 
functionally interrelated to the medium and will be statutory in most cases since use of technology permits the 
function of the descriptive material to be realized. Compare In re Lowry, 32 F.3d 1579, 1583-84, 32 USPQ2d 
1 03 1 , 1 035 (Fed. Cir. 1 994) (claim to data structure stored on a computer readable medium that increases 
computer efficiency held statutory) and Warmerdam, 33 F.3d at 1360-61, 31 USPQ2d at 1759 (claim to 
computer having a specific data structure stored in memory held statutory product-by-process claim) with 
Warmerdam, 33 F.3d at 1361, 31 USPQ2d at 1760 (claim to a data structure per se held nonstatutory). 

In contrast, a claimed computer-readable medium encoded with a computer program is a computer element 
which defines structural and functional interrelationships between the computer program and the rest of the 
computer which permit the computer program's functionality to be realized, and is thus statutory. See Lowry, 32 
F.3d at 1 583-84, 32 USPQ2d at 1 035 . 



2. Claims 1-10 and 12-15 are rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter as follows. Claim 1 defines in a console processing unit 
for goods inventory management coupled via the Internet to at least one fixed detector and at 
least one mobile sensor, a data structure for representing a monitored object, the data structure 
comprising embodying functional descriptive material. However, the claim does not define a 
computer-readable medium or memory and is thus non-statutory for that reason (i.e., "When 
functional descriptive material is recorded on some computer-readable medium it becomes 
structurally and functionally interrelated to the medium and will be statutory in most cases since 
use of technology permits the function of the descriptive material to be realized" - Guidelines 
Annex IV). That is, the scope of the presently claimed data structure can range from paper on 
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which the program is written, to a program simply contemplated and memorized by a person. 
The examiner suggests amending the claim to embody the program on "computer-readable 
medium" or equivalent in order to make the claim statutory. Any amendment to the claim 
should be commensurate with its corresponding disclosure. 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in" the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

4. Claims 1-3, 5-12, 14-16, 18-21, and 23 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Rauber et al (US 6,182,053) in view of Woolston (US 5,845,265). 

Re claims' I, 3, 5, 7, 11-12, 14, 16, 18, 20-21, and 23, Rauber teaches in a console 
processing unit (12 of fig. 1) for goods inventory management coupled via LOCAL- AREA- 
NETWORK (LAN) connection, at least one mobile sensor ( 1 6 of fig. 1 ), 

a data structure for representing a monitored object (freight), the data structure a 
processor (12 of fig. 1) provided in a wireless target unit (16 of fig. 1; 18 of fig. 1) for 
transmitting or receiving in a network a data structure (LAN), comprising: an object identifier 
(12 of fig. 1; see also 302-308 of fig. 3) representing one or more goods in production, inventory 
and shipment (col. 7, lines 1-63); a first object location and a time monitored at such location 
(212-218 of fig. 2; Note the warehouse would obviously have a detector for detecting the first 
object (good, product, merchandize)), provided by a detector coupled to the console processing 
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unit (12 of fig. 1); and a second object location and a time monitored at such location (220 of fig. 
2, Note the retail store would obviously have scanner for scanning the second object (good, 
product, merchandize)), provided by a sensor coupled to the console processing unit (12 of fig. 
1); wherein an access means (12 or 18 of fig. 1; Col. 8, lines 6-13) processes the data structure 
securely using a digital certificate, watermark or encryption key (Note enter his or her 
salesperson ID number and customer ID number), such that the data structure is accessible for 
object-monitoring from only one or more specified network site (12, 18 of fig. 1) or processors 
the data structure being provided automatically using control software (fig. 2) for network 
surveillance in response to a user search query (col. 8, lines 29-44); the software (fig. 2) 
comprising: a network and data communication module (col. 8, lines 45-52; Note any of the RF 
unites would be used to communicate for the inventory method); an object and map database 
(col. 7, lines 30-36); an object movement processing module (308 of fig. 3; Note updating the 
object location; where in step 308, the computer program prompts the operator to enter the 
location within the distressed inventory warehouse where the inventory will be initially stored 
before it proceeds through further stages of the inventory management method. As described in 
more detail below, this location may include a warehouse area, a retail area, or other specialized 
areas for particular types of freight); 

a security management module (col. 8, lines 6-13); an electronic transaction processing 
modules (col. 8, lines 53-61); a diagnosis tool (504 of fig. 5); a performance report updater 
modules (508 of fig. 1); and a visual object analyzer module (col. 8, lines 62-67); whereby such 
modules are functionally integrated to enable surveillance-based commercial transaction using 
the data structure (figs. 1 and 2); wherein the sensor (16 of fig. 1) comprises a radio-frequency 
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identification device for locating the identified goods (16 of fig. 1); wherein a neural network or 
simulation program for recognizing adaptively on or more identified goods for real-time tracking 
a multiple goods movement (col. 10, lines 5-28, Note the inventory management would 
recognized the multiple goods that have been selling from one of the PGU's, where the inventory 
management would obviously determines how many items that are available in the warehouse 
and in the store, e.g. Wal-mart, using dual two-gigar byte mirroring hard drives, and an 
integrated local-area-network (LAN) connection with the host computer (12 of fig. 1) may be 
any type of computer including a microcomputer, minicomputer, or mainframe computer). 

It is noted that Rauber does not particularly teach Internet and at least one fixed detector 
that comprises a camera for observing such identified goods, thereby enabling the sensor and the 
detector to provide surveillance of the identified goods within an observable range in which the 
sensor is mobile relative to the detector; the detector comprises visual-analyzer means for 
recognizing adaptively the identified goods using a neural network or simulation program, 
thereby enabling secure inventory management of the identified goods as claimed. 

However, Rauber suggests that equivalents may be employed and substitutions made 
herein, so this is evidence to one skilled in the art to modify any conventional and suitable 
devices and methods to the Rauber' s system. 

Woolston teaches Internet (Col. 14, lines 51-63) and at least one fixed detector (12 of fig. 
1) that comprises a camera (12 of fig. 1) for observing such identified goods, thereby enabling 
the sensor (14 of fig. 1) and the detector (12 of fig. 1) to provide surveillance of the identified 
goods within an observable range (the camera 12 of figure 1 is able to capturing the goods within 
an observation range) in which the sensor (14 of fig. 1) is mobile relative to the detector; the 
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detector comprises visual-analyzer means (920 of fig. 13, viewing goods) for recognizing 
adaptively the identified goods using a neural network or simulation program (the image is 
generated by the camera (12 of fig. 1) and is displaying on the display (16 or 28 of fig. 1)), 
thereby enabling secure inventory management of the identified goods (804 of fig. 8; Note it is 
understood that a secure and/or encrypted means may be established between a participant's 
interface application and a consignment node to transfer sensitive or theft prone information); a 
software agent (the computer (10 of fig. 1) would obviously have a software agent associated 
with the monitored object for accessing a database (10 of fig. 1, database for accessing 
network)); the observable range is modifiable according to a rule set (far or near the camera (12 
of fig. 1)), wherein network is used for the inventory management is provided (fig. 1) and 
tracking down where the items or goods are located for inventory (the operation would 
obviously track or view the goods for inventory). 

Therefore, taking the teachings of Rauber and Woolston as a whole, it would have been 
obvious to one of ordinary skill in the art to incorporate the teachings of Woolston into the 
console processing unit of Rauber for observing goods with its price during inventory. Doing 
would allow the user to easily set up his or her own warehouse, store, or retailer for buying and 
selling goods via the Internet. 

Re claim 2, Rauber further teaches a scheduled object location and a time scheduled for 
such location (col. 7, lines 49-63). 

Re claim 6, Rauder further teaches the object identifier (12 of fig. 1) comprises an object 
name, an object group, an object query, an object condition, an object status, an object location, 
an object time, an object error, or an object image, video, or audio broadcast signal. 
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Re claim 8, Rauber further teaches the monitored object is monitored temporarily using 
an extrapolated or last-stored positional or visual signal (506 of fig. 5). 

Re claim 9, Rauber further teaches the monitored object is authenticated according to a 
voice pattern, a finger-print pattern, a handwritten signature, or a magnetic or smart-card signal 
(fig. 4, ID salesman). 

Re claim 1 0, Rauber further teaches the monitored object is provided an electronic file 
comprising a book, a greeting card, a news report, a sports report, a stock report, an artwork, a 
research database, a personal list, a recorded or live voice or music transmission, an electronic 
tool, or a commercial transaction (224, 226, and 228 of fig. 2). 

Re claims 1 5 and 19, Rauber further teaches the data structure indicates in-stock 
availability of the identified goods for transacting shipment (fig. 3), and a tax-rate for transaction 
at the location of the identified goods (426 of fig. 4). 

5. Claims 13,17, and 22 are rejected under 35 U.S.C. 1 03(a) as being unpatentable over 
Rauber et al (US 6,182,053) in view of Woolston (US 5,845,265) as applied to claims 1, 1 1, and 
20, and further in view of Durbin et al. (US 6,039,258). 

Re claims 13, 17, and 22, the combination of Rauber and Woolston teaches the sensor is 
mobile in the observation range except a sensor signal port for sensing a low-power or fuel 
condition of the identified goods, thereby enabling the console processing unit to indicate or 
warm a down period for using the identified goods as claimed. 

Durbin teaches a sensor signal port for sensing a low-power or fuel condition of the 
identified goods, thereby enabling the console processing unit to indicate or warm a down period 
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for using the identified goods (14 of fig. 1 ; Note the somatic communication system 14 may be 
utilized to indicate when the battery level is low, when there is a system error in the operating 
system, and other various alarm type of signals). 

Therefore, taking the teachings of Rauber, Woolston, and Durbin as a whole, it would 
have been obvious to one of ordinary skill in the art to incorporate the teachings of Durbin into 
the combined system of Rauber and Woolston for recognizing the low power or battery of the 
sensor as suggested by Durbin (14 of fig. 1). Doing so would allow the user to change or 
recharge the battery in advance to prevent damage of the handheld or portable sensor. 

1 . Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over Rauber et al (US 
6,182,053) in view of Woolston (US 5 ; 845,265) as applied to claim 1, and further in view of 
Kennedy (US 6,301,480). 

Re claim 4, the combination of Rauber and Woolston does not particularly teach the 
detector comprises an accelerometer as claimed. 

However, Kennedy teaches a mobile communication unit (12 of fig. 1) comprises an 
accelerometer and personal health sensor (col. 3* lines 5-19). 

Therefore, taking the combined teachings of Rauber, Woolston and Kennedy as a whole, 
it would have been obvious to one of ordinary skill in the art to incorporate the teachings of 
Kennedy into the system of the combined system of Rauber and Woolston for the same purpose 
of communicating between the remote buyer and central station fast and more accuracy. 
Doing so would provide the advantages of the system include the adaptation of the system to 
provide mobile units are associated with cars, trucks, boats, barges, airplanes, cargo holders, 
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persons or other mobile items such as ambulance vehicle that desire a selection of services. 
These services include emergency services, roadside assistance, information services (e.g., 
directions, news and weather reports, financial quotes, etc.), or other as suggested by Kennedy. 

Conclusion 

2. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

Odom et al (US 6,058,379) discloses real-time network exchange with seller specified 
exchange parameters and interactive seller. 

Contact Information 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Tung Vo whose telephone number is 571-272-7340. The 
examiner can normally be reached on Monday-Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mehrdad Dastburi can be reached on 571-272-7418. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272J0.00... . 

Tung Vo 
Primary Examiner 
Art Unit 2621 
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